Gerrit's Blog

How to create a simplistic blog for your terminal

2026-02-08T00:00:00Z

You might already know about or have used things like wttr.in or cheat.sh. There is a ton more you can explore on this list when it comes to accessing services from the shell (awesome-console-services) . All those sites can be visited with curl and provide a terminal-centric experience. Of course this nerd-sniped me enough to try this out for my own blog.

There are are three things that I need to accomplish while implementing this:

Parse the same blog posts from sources as the original blog
Introduce colorization not only for blog posts but also the index page
Combine all above and put the output in a target directory
Make the blog server seamless provide the terminal compatible pages instead of the HTML page

Parse pages for the blog

As described in a previous post the blog pages are written in Markdown with metadata provided as Front Matter which allows to add some metadata to the files, like this

---
layout: post
file_name: blog_infrastructure
title: "Maintain a simplistic blog"
description: "Hacking around to be lazy later"
date: 2023-08-07
categories: git obsidian ios
---

So this leaves me with two problems. First extract the metadata for generating the file names and the additional information like date and categories and title. After this render the whole content to a new file which has render highlights.

Using `yq` for metadata extraction

Taking the Front Matter example from above, let's see what yq can extract.

> yq --front-matter=extract 'Maintain a simplistic blog.md'
---
layout: post
file_name: blog_infrastructure
title: "Maintain a simplistic blog"
description: "Hacking around to be lazy later"
date: 2023-08-07
categories: git obsidian ios

Which looks pretty much like the block that is in the Markdown source.

To get the individual fields, their names can be added to the command.

> yq --front-matter=extract ".file_name" 'Maintain a simplistic blog.md'
blog_infrastructure

With this I have all at hand to generate the correct output files and use the metadata for the index page.

Render the blog post

For rendering I had two options in mind: bat (https://github.com/sharkdp/bat) or glow (https://github.com/charmbracelet/glow/). Not because I did a great research but because both are already installed on my machine. After tinkering around with bat first, I decided to go with glow, because I couldn't create output that contained ANSI colors to use it later.

But also with glow it was not straight forward to get the output as expected. There are two mandatory environment variables that needs to be set when invoking the command: CLICOLOR_FORCE=1 and COLORTERM=truecolor Which will result in "a little bit" overhead for the colors and font-style when produced than you would add manually, but maybe this is the pill to swallow when using generators.

ESC[94;1mESC[0mESC[94;1mESC[0m  ESC[94;1m## ESC[0mESC[94;1mDisclaimerESC[0mESC[37mESC[37m ESC[0mESC[37m ESC[0mESC[37m ESC[0mESC[37m ESC[0mESC[37m ESC[0mESC[37m ESC[0mESC[37m ESC[0mESC[37m ESC[0mESC[37m ESC[0mESC[37m ESC[0mESC[37m ESC[0mESC[37m ESC[0mESC[37m ESC[0mESC[37m ESC[0mESC[37m ESC[0mESC[37m ESC[0mESC[37m ESC[0mESC[37m ESC[0mESC[37m ESC[0mESC[37m ESC[0mESC[37m ESC[0mESC[37m ESC[0mESC[37m ESC[0mESC[37m ESC[0mESC[37m ESC[0mESC[37m ESC[0mESC[37m ESC[0mESC[37m ESC[0mESC[37m ESC[0mESC[37m ESC[0mESC[37m ESC[0mESC[37m ESC[0mESC[37m ESC[0mESC[37m ESC[0mESC[37m ESC[0mESC[37m ESC[0mESC[37m ESC[0mESC[37m ESC[0mESC[37m ESC[0mESC[37m
ESC[0mESC[37m ESC[0mESC[37m ESC[0mESC[37m ESC[0mESC[37m ESC[0mESC[37m ESC[0mESC[37m ESC[0mESC[37m ESC[0mESC[37m ESC[0mESC[37m ESC[0mESC[37m ESC[0mESC[37m ESC[0mESC[37m ESC[0mESC[37m ESC[0mESC[37m ESC[0mESC[37m ESC[0mESC[37m ESC[0mESC[37m ESC[0mESC[37m ESC[0mESC[37m ESC[0mESC[37m ESC[0mESC[37m ESC[0mESC[37m ESC[0mESC[37m ESC[0mESC[0m

("renderable" output for the first lines which only produce an header, replace the ESC with \x1b commands mentally)

The colored index page

This is all based on the idea of using ANSI color escapes. While I was looking for more information, I stumbled over this nice stackoverflow answer, which contains even more information on text styling and navigation boiled down from the Wikipedia ANSII escape code page.

Short ANSI color command detour

But let's start simple. If you jumped to the linked answer above already, this is nothing new, but here is a quick excerpt on the usage.

Let's say you want to display a text in the terminal in red. This would be the definition of

\x1b - escape sequence (can also be \033)
31m  - color for red foreground

While this is taken from the original specification which only had 8 colors. I am aiming big here and want to choose from the whole 8-bit table ;) This way the command gets a little bit more complex

\x1b  - escape sequence (can also be \033)
38;5; - we want to set the foreground color
9m    - basic color value for red

If you want to use even more, you can also go full 24-bit and define every value individual by using ESC[38;2;⟨r⟩;⟨g⟩;⟨b⟩m which would be ESC[38;2;255;0;0m for red.

As you have already seen in the Markdown output, it's always a good idea to make use of the reset command sequence after the color -or any other- definition: ESC[0m.

Using colors

With this, I could define a small color reference in my script.

DATE_COLOR=$'\x1b[38;5;226m'
TITLE_COLOR=$'\x1b[38;5;211m'
LINK_COLOR=$'\x1b[38;5;14m'
CATEGORIES_COLOR=$'\x1b[38;5;32m'
RST=$'\x1b[0m'

And later reference it to style my texts

format_title() {
	echo -n "${TITLE_COLOR}$1${RST}"
}
post_title=$(yq --front-matter=extract ".title" "$n")
format_title "$post_title"

Automate and build

As you have already guessed by the examples, the whole build is meant to be executed in the terminal with some command line tools. The basic idea is to prepare an index page with some static content / "logo" and then loop over the Markdown sources, convert them to text files, and as a side effect extract the metadata as text for the post list. All this could be invoked by a simple shell script, but let's get a little bit nerdy here and use just here to have a nicer invocation.

My justfile looks like this

clear:
        rm -rf output

build:
        mkdir output
        ./create-header.sh > output/index
        ./add-posts.sh $POST_SOURCES output >> output/index

The benefit of using just is to make use of environment variables, in contrast to a Makefile.

As already mentioned, the create-header.sh is pretty much simple, so assume it just creates the index file. The most work is done in the add-posts.sh:

#!/usr/bin/env bash

SOURCE_DIR=$1
OUTPUT_DIR=$2

DATE_COLOR=$'\x1b[38;5;226m'
TITLE_COLOR=$'\x1b[38;5;211m'
LINK_COLOR=$'\x1b[38;5;14m'
CATEGORIES_COLOR=$'\x1b[38;5;32m'
RST=$'\x1b[0m'

format_title() {
	echo -n "${TITLE_COLOR}$1${RST}"
}
format_date() {
	echo -n "${DATE_COLOR}$1${RST}"
}
format_link() {
	echo -n "${LINK_COLOR}$1${RST}"
}
format_categories() {
	echo -n "${CATEGORIES_COLOR}$1${RST}"
}

while IFS= read -r -d '' n; do
	post_categories=$(yq --front-matter=extract ".categories" "$n")
	post_title=$(yq --front-matter=extract ".title" "$n")
	file_name=$(yq --front-matter=extract ".file_name" "$n")
	post_date=$(yq --front-matter=extract ".date" "$n")
	read -r -a post_cat_array <<< "$post_categories"
	cat "$n" | CLICOLOR_FORCE=1 COLORTERM=truecolor glow --style dark --width 120 > $OUTPUT_DIR/$file_name
	formatted_date=$(format_date $post_date)
	formatted_title=$(format_title "$post_title")
	formatted_link=$(format_link $file_name)
	formatted_categories=$(format_categories "$post_categories")
	entries+="$formatted_date - $formatted_title - /$formatted_link ($formatted_categories)"
	entries+=$'\n'
done < <(find "$SOURCE_DIR" -iname "*.md" -type f -print0)
entries_sorted=$(echo -n "$entries" | sort -r)
echo "${entries_sorted}"

I iterate over all markdown files in the $SOURCE_DIR and, as explained above, extract all the metadata, convert the markdown file (the glow command invocation) and add the information to an entries list to be displayed in the index page. This list gets sorted by date, yes this is why it's the first bit in the item string, and will be echoed to be added to the index page.

And with all of this, the text-only blog will be put into the output directory.

Serving the curl clients

The content will be uploaded into a subdirectory called curl on the web server. Now the only thing left to do is to configure the server (nginx) to serve those pages instead of the HTML version of the blog.

This is pretty much easy to do because there is a directive in nginx which allows us to check for the User-Agent provided by the client. Whenever a user now calls my blog using the an User-Agent containing "curl", the request will be rewritten to look into the curl subdirectory (of the server definition's root directory). This location will serve the content as it would serve the html pages.

location / {
	if ( $http_user_agent ~* 'curl' ) {
		rewrite ^/(.*)$ /curl/$1 last;
	}
	try_files $uri $uri/ =404;
}
location /curl {
	index index;
	try_files $uri $uri/ =404;
}

Calling this blog now with curl 'https://meistermeier.com' returns now

                _     _                           _
               (_)   | |                         (_)
 _ __ ___   ___ _ ___| |_ ___ _ __ _ __ ___   ___ _  ___ _ __
| '_ ` _ \ / _ \ / __| __/ _ \ '__| '_ ` _ \ / _ \ |/ _ \ '__|
| | | | | |  __/ \__ \ ||  __/ |  | | | | | |  __/ |  __/ |
|_| |_| |_|\___|_|___/\__\___|_|  |_| |_| |_|\___|_|\___|_|



2024-12-28 - Spring Actuator Security - /spring-actuator-security (java spring)
2024-11-01 - Enable Cypher Language Server for Neovim - /cypher_ls_neovim_part2 (neovim cypher neo4j)
2024-02-23 - Spring AI with Neo4j Vector Index - /spring-ai-neo4j (neo4j ai cypher)
2023-12-04 - Cypher Language Server support for Neovim - /cypher_ls_neovim (neovim cypher neo4j)
2023-08-07 - Maintain a simplistic blog - /blog_infrastructure (git obsidian ios)
2023-07-06 - First experiences with DuckDB (and SQL in general after a while) - /duckdb-part1 (duckdb sql)
2023-06-22 - Using Spring for GraphQL With Spring Data Neo4j - /sdn-graphql-1 (graphql spring neo4j)

and using e.g. the first link in the list, invoking curl (with -s for silent) again and passing it on to e.g. less with the -R flag to preserve colors (curl -s 'https://meistermeier.com/spring-actuator-security' | less -R) gives a readable version for consuming the content on the terminal with nice formatting.

Was it needed? Definitely not, but it was so much fun to just hack some nerdy things together. From the idea to execution it might have taken less than writing this post. And why not also add this option to your page ;)

Spring Actuator Security

2024-12-28T00:00:00Z

Yesterday I was looking at this great talk about the leak of private information at Volkswagen. (Recordings will be found here https://media.ccc.de/c/38c3) Being a Java and Spring developer it made me a little bit sad that the part of the talk about the initial steps involving the Spring Boot based application sounded like all applications using the actuator are insecure by design. Yes, I can completely get that there are time constraints for such a talk and you also want to get your jokes right, and I am the last one to judge about this because I felt entertained. But let's have a more detailed look what it takes to get the heapdumps of a running Spring Boot application, how to explore them for interesting data, and -last but not least- how to secure such endpoints.

Setup

The example application does not require a huge setup. As always if I want to quickly create a Spring Boot application, I go to https://start.spring.io. The dependencies needed are the actuator and the web dependency, foremost to keep the application running.

After downloading and extracting the archive, the application can be started by invoking

./mvnw spring-boot:run

When accessing now the actuator endpoint of the application, the active ones are listed.

curl 'http://localhost:8080/actuator'

{
  "_links": {
    "self": {
      "href": "http://localhost:8080/actuator",
      "templated": false
    },
    "health-path": {
      "href": "http://localhost:8080/actuator/health/{*path}",
      "templated": true
    },
    "health": {
      "href": "http://localhost:8080/actuator/health",
      "templated": false
    }
  }
}

If you came here because you were afraid that every application now has a heapdump endpoint available by default, I am happy to tell you that this is not the case.

The obvious next question is: How to get the heapdump endpoint enabled?

Enabling the heapdump endpoint

A lot of properties of a Spring Boot application can be set via the application.properties configuration (or similar ways). There is not only so much built-in but also a very good documentation about all the properties being available.

Let's activate the endpoint for the heapdump explicitly. The property that needs to be set is

management.endpoints.web.exposure.include=heapdump

After restarting the application and checking back the available endpoints, this one is now activated.

{
  "_links": {
    "self": {
      "href": "http://localhost:8080/actuator",
      "templated": false
    },
    "heapdump": {
      "href": "http://localhost:8080/actuator/heapdump",
      "templated": false
    }
  }
}

As you can see, there is now only one endpoint for the heapdump and the health endpoint(s) disappeared. You might have guessed it, the property above takes a list of endpoints to activate.

Adding some credentials

To get something into the application that reflects a scenario where some sensitive data can be found, a class will be added that holds those data.

public class CloudProviderXyz {  
      
    private final String username;  
      
    private final String password;  
  
    public CloudProviderXyz(String username, String password) {  
        this.username = username;  
        this.password = password;  
    }  
}

To create an instance of this, we add a @Bean definition in the application class.

@SpringBootApplication  
public class ActuatorDemoApplication {  
      
    @Bean  
    public CloudProviderXyz cloudProviderXyz() {  
       return new CloudProviderXyz("meistermeier", "myverysecretpassword");  
    }  
  
    public static void main(String[] args) {  
       SpringApplication.run(ActuatorDemoApplication.class, args);  
    }  
  
}

After this, the application gets restarted and is ready to be inspected.

Heapdump analysis

With the actuator's heapdump endpoint enabled, it is now possible to trigger a heapdump creation and download it.

curl 'http://localhost:8080/actuator/heapdump' -O

After downloading the file, it can be opened by any heapdump tool, e.g. VisualVm, which will be used for this example.

The overview page should look similar to this

Let's switch to the Objects view to inspect the instances alive during the time the heapdump was taken.

Here we can search for an instance of our newly created class CloudProviderXyz. And of course with this, also its fields and their current content.

But what could be done to avoid those data being accessible to everyone?

Bytes and obfuscation

Assuming that the credential storing class is really under our control, could the sensitive data be stored different, e.g. in an byte array to be safer? Short story short: No. Of course the byte values are readable the same way.

Basically this is valid for all obfuscation methods and reflects the typical security guideline: Depending on how much you make it complicated by obfuscation, the longer it will just take to reach the desired data. Even if this data is only used for a second and the enclosing object gets removed immediately, there is still the chance that the timing is right to get those sensitive data via the heapdump. It will never be 100% secure.

Externalizing the credentials

Because it is necessary to say, I assume, this won't protect the credentials within your application. But besides this, you should always externalize credentials/secrets/... using e.g. Vault or similar products. No one should ever have plain text access to those sensitive data besides the application at the time it needs it.

Securing the endpoint

As you have may already concluded, the moment your application needs some kind of secrets/credentials it will have it. If there is any need to have the feature of heapdump enabled in a production environment, it should be secured. The whole actuator feature is well documented and also mentions the needed techniques to secure the access.

Spring Security is a very mature project in the Spring ecosystem and the de-facto default when it comes to authorization and access control within Spring application. Just add its Spring Boot dependency

<dependency>  
    <groupId>org.springframework.boot</groupId>  
    <artifactId>spring-boot-starter-security</artifactId>  
</dependency>

and configure a default security mechanism for all actuator endpoints.

@Bean  
public SecurityFilterChain securityFilterChain(HttpSecurity http) throws Exception {  
    http.securityMatcher(EndpointRequest.toAnyEndpoint());  
    http.authorizeHttpRequests((requests) -> requests.anyRequest().authenticated());  
    http.httpBasic(Customizer.withDefaults());  
    return http.build();  
}

This is a simplified chain, but it can be configured to e.g. match only specific endpoints, require some group membership, or specific rights. But this would be beyond this overview.

When the application gets restarted, two things can be noted. First, there is a note about a created password:

2024-12-28T10:26:00.072+01:00  WARN 78599 --- [actuator-demo] [           main] .s.s.UserDetailsServiceAutoConfiguration : 

Using generated security password: ad882bb1-fcb3-4378-875b-1ad63c32997d

This generated password is for development use only. Your security configuration must be updated before running your application in production.

And secondly, if the heapdump endpoint gets now queried, it will deny access.

curl 'http://localhost:8080/actuator/heapdump'    
{"timestamp":"2024-12-28T09:26:06.738+00:00","status":403,"error":"Forbidden","path":"/actuator/heapdump"}

Providing the (default) username user and the generated password as HTTP basic auth parameter to the curl command

curl -u user:ad882bb1-fcb3-4378-875b-1ad63c32997d 'http://localhost:8080/actuator/heapdump'

will give us access to the endpoint. Of course as mentioned above this security setup should be hardened more and the password generator should only to be used for development.

I tried to give an overview here to put the mentioned "flaw" in Spring Boot / Actuator in some perspective and context. If there is anything, I missed, I would be happy if you ping me at https://mastodon.social/@meistermeier

Enable Cypher Language Server for Neovim

2024-11-01T00:00:00Z

Update 13.11.2014: Add missing config files and adjust directories.

This is the second part of my post about Cypher support in Neovim. If you missed the first one that shows the evolution of adding support, you're welcome to read the first part. In this post I will focus on getting a Neovim environment from 0 to Cypher support (also with a pleasant visual improvement). This guide assumes that you don't have anything configured right now. If you already have a plugin manager enabled or even language server plugins installed, you can happily skip those sections.

Requirements

There is not much that is needed to get the full Cypher support working. The only thing needed, besides Neovim obviously, is a current npm and nodejs version for the Cypher language server to be installed and run.

Setting up the needed infrastructure

First ensure that your Neovim installation is on a version that has Cypher file support (v0.10.0+) by calling nvim -v. The output should contain something like

NVIM v0.10.2
Build type: Release

When opening a .cypher file it should look like this:

Because this guide starts with a fresh configuration, we need to add some boilerplate code to get a package manager running in Neovim. I have chosen to use lazy.nvim and something similar to its default setup. Which leads to a ~/.config/nvim/init.lua like this,

require("config.lazy-bootstrap")

vim.g.mapleader = " "
vim.g.maplocalleader = "\\"

require("config.lazy")

one file called ~/.config/nvim/lua/config/lazy-bootstrap.lua for the initial installation of the package manager, if it does not exist,

-- Bootstrap lazy.nvim
local lazypath = vim.fn.stdpath("data") .. "/lazy/lazy.nvim"
if not (vim.uv or vim.loop).fs_stat(lazypath) then
  local lazyrepo = "https://github.com/folke/lazy.nvim.git"
  local out = vim.fn.system({ "git", "clone", "--filter=blob:none", "--branch=stable", lazyrepo, lazypath })
  if vim.v.shell_error ~= 0 then
    vim.api.nvim_echo({
      { "Failed to clone lazy.nvim:\n", "ErrorMsg" },
      { out, "WarningMsg" },
      { "\nPress any key to exit..." },
    }, true, {})
    vim.fn.getchar()
    os.exit(1)
  end
end
vim.opt.rtp:prepend(lazypath)

and one called ~/.config/nvim/lua/config/lazy.lua for the initialization of the plugin manager including all needed plugins.

require("lazy").setup({
  spec = {
  -- here we will place the plugins
  },
  install = { colorscheme = { "habamax" } },
  checker = { enabled = true },
})

When Neovim gets started now, we should see...nothing. This is due to the fact that there is no plugin defined yet, let's change this.

Adding the needed plugins

To get the right language server support into Neovim, we need three plugin:

nvim-lspconfig - required for having built-in LSP support (filetype association, start/stop, etc.)
mason - handles the installation of LSPs, linters, etc.
mason-lspconfig - bridges nvim-lspconfig with maven to have auto-installation etc. in place

More details on what those plugins do can be found in my older blog post .

Because the targeted setup is straight forward, those plugins can be added directly to the spec section in the lazy setup:

require("lazy").setup({
  spec = {
	  "neovim/nvim-lspconfig",
	  "williamboman/mason-lspconfig.nvim",
	  "williamboman/mason.nvim"
  },
  install = { colorscheme = { "habamax" } },
  checker = { enabled = true },
})

In the next step, another config file (~/.config/nvim/lua/config/mason.lua) gets added that will enable mason and tell mason-lspconfig which language server to install when asked for.

require("mason").setup {}
require("mason-lspconfig").setup {
    ensure_installed = { "cypher_ls" },
}

And enable the language server in the init.lua.

require("config.lazy-bootstrap")

vim.g.mapleader = " "
vim.g.maplocalleader = "\\"

require("config.lazy")
require("config.mason")

require('lspconfig')['cypher_ls'].setup {}

When opening a .cypher file now, it will not just highlight the syntax but also show information on wrong usage.

Make it even nicer (optional and opinionated)

The current color scheme is using the default shell colors. It depends on the contrast, if the individual parts are recognizable. You know what makes the world a better place and also your Neovim and everything else much more enjoyable? The Catppuccin color palette. Luckily there is already a Neovim plugin that will help with the setup.

First the catpuccin Neovim plugin needs to be added to the list of plugins.

require("lazy").setup({
  spec = {
	  "neovim/nvim-lspconfig",
	  "williamboman/mason-lspconfig.nvim",
	  "williamboman/mason.nvim",
	  { "catppuccin/nvim", name = "catppuccin", priority = 1000 }
  },
  install = { colorscheme = { "habamax" } },
  checker = { enabled = true },
})

The syntax looks a bit different for this plugin, but this is mostly due to the fact that otherwise it would have the default name nvim, what's for sure not the intention. The priority (default 50) just says that this plugin should be loaded as late as possible on start, because it is not important for other plugins.

Having done this, let's get back to the init.lua and define the color scheme.

require("config.lazy-bootstrap")

vim.g.mapleader = " "
vim.g.maplocalleader = "\\"

require("config.lazy")
require("config.mason")

require('lspconfig')['cypher_ls'].setup {}

vim.cmd.colorscheme "catppuccin"

Et voilà, we have a much nicer color scheme. Head over to the color scheme's documentation to get a taste for all the flavours that are offered ;)

Happy Cypher writing...

Spring AI with Neo4j Vector Index

2024-02-23T00:00:00Z

SpringAI

At the moment still in pre-release phase, there is a new Spring module called SpringAI. It offers support for different "AI-vendors" like OpenAI, Bedrock, Ollama, etc. for creating embeddings or calling the chat completion feature. (Yes, also image creation, but this is not part of this post.) On the other end there are multiple store implementations to persist the embedding in various stores (PgVector, Redis, Neo4j etc.).

The idea of those store modules is to have a unified idea of storing Documents representing some content along with metadata into the store and adding the embeddings. Also, the stores offer a database specific implementation of a similarity search. The result of the search is -described on a high abstraction level- the documents having an embedding "close" to the given term provided with the similarity search invocation.

Since the project is very actively developed, it is best to check out the current documentation for more information.

Neo4j Vector Store

In this post, we will have a closer look into the usage of the Neo4j Vector Store implementation created by Michael Simons and me. As mentioned in the beginning, each store module unifies the behaviour defined by the VectorStore interface, so does the Neo4j implementation.

The store itself is configurable for your needs. By default, it will store the Documents as nodes labelled Document and store the embeddings on the embedding property. For customisation, it is possible to define label, property name for the embedding, index name, embedding dimension, or distance type for the index. When doing the first steps with SpringAI this is not needed but might come in handy if you have an already existing dataset within the database and you don't want to be forced into the default naming schema.

After initializing the Neo4j Vector Store, it will ensure that a vector index for the embedding exists. At the moment of writing, Neo4j supports embedding dimensions up to 2048. As a consequence retrieving embeddings e.g. via llama2 does not work because they have an embedding dimension of 4096.

Adding an embedding to a node will set the embedding by calling db.create.setNodeVectorProperty(node, <embedding_property>, <embedding>). This way, it is guaranteed that the value will get indexed.

Retrieving the Documents via similarity search will invoke db.index.vector.queryNodes(<index_name>, <amount_of_nearest_neighbours>, <search_term_as_embedding>), and return the matching nodes alongside their ranking.

Example

This example will not only demonstrate how to use the functionality mentioned above, but also how to create some Documents, and use them after a successful similarity search with the chat completion to get the information, you are looking for. The application should be a simple Spring Boot web application that accepts a user prompt and responds with the equivalent Cypher statement. We won't strive for perfection here but give some idea how you could also create something similar for your purposes.

Setup

A Neo4j instance version 5.15 or newer is required to follow along. If you don't know how to get one, there are several options to get one running:

We will create a Spring Boot application at start.spring.io with the -at the moment- current version 3.2.3 and select only the web dependency (pre-configured application). Because the SpringAI project is still in pre-release phase, it is only available in Spring's repositories. After downloading and extracting the application, the additional repositories needs to get declared in the pom.xml.

<repositories>  
    <repository>  
        <id>spring-milestones</id>  
        <name>Spring Milestones</name>  
        <url>https://repo.spring.io/milestone</url>  
        <snapshots>  
            <enabled>false</enabled>  
        </snapshots>  
    </repository>  
    <repository>  
        <id>spring-snapshots</id>  
        <name>Spring Snapshots</name>  
        <url>https://repo.spring.io/snapshot</url>  
        <releases>  
            <enabled>false</enabled>  
        </releases>  
    </repository>  
</repositories>

There is a BOM (bill of materials) available to define the overall version of SpringAI modules.

<dependencyManagement>  
    <dependencies>  
        <dependency>  
            <groupId>org.springframework.ai</groupId>  
            <artifactId>spring-ai-bom</artifactId>  
            <version>${spring-ai.version}</version>  
            <type>pom</type>  
            <scope>import</scope>  
        </dependency>  
    </dependencies>  
</dependencyManagement>

In this example, we will use the version 0.8.0-SNAPSHOT.

Now we can add the needed modules of SpringAI for this example:

<dependency>  
    <groupId>org.springframework.ai</groupId>  
    <artifactId>spring-ai-neo4j-store-spring-boot-starter</artifactId>  
</dependency>  
<dependency>  
    <groupId>org.springframework.ai</groupId>  
    <artifactId>spring-ai-openai-spring-boot-starter</artifactId>  
</dependency>  
<dependency>  
    <groupId>org.springframework.ai</groupId>  
    <artifactId>spring-ai-pdf-document-reader</artifactId>  
</dependency>

As you can see, the store and AI vendor specific dependencies are offered as a Spring Boot starter. This means that we don't need to do any manual bean configuration but just provide the needed properties. If we are ok with the default settings of the store and OpenAI module (we are), there are just some mandatory properties left, that should be placed in the application.properties file:

spring.ai.openai.api-key=${OPENAI_API_KEY}  
  
spring.neo4j.uri=<neo4j_URI> 
spring.neo4j.authentication.username=<neo4j_username>  
spring.neo4j.authentication.password=<neo4j_password>

The OpenAI API token and the coordinates of the Neo4j database to store and query the Documents are required.

The third dependency is a PDF to Document converter module that is part of SpringAI to simplify the tokenisation of PDF documents into Document objects.

The application infrastructure is set up and we can start coding.

Application

The application should not only return Cypher statements for textual input but also trigger the creation of the documents from a given PDF file.

Controller

Let's start with the controller class CypherPromptController that we declare as a @Controller stereotype and add the @ReponseBody annotation to be able to return basic strings. The actual functionality of invoking the vector store and AI services will be done in a service class called CypherPromptService. This needs to get autowired into our controller. There are two endpoint that needs to be provided, one for the Document and embedding creation and one for the actual prompt to Cypher conversion.

@Controller  
@ResponseBody  
public class CypherPromptController {
  private final CypherPromptService promptService;  
  
  public CypherPromptController(CypherPromptService promptService) {  
    this.promptService = promptService;  
  }
  @GetMapping("/")  
  public String convertToCypher(@RequestParam("m") String message) {  
    return this.promptService.convertToCypher(message);  
  }  
  
  @GetMapping("/create")  
  public String createDocuments(@RequestParam("filePath") String filePath) {  
    this.promptService.createEmbeddings(filePath);  
    return "done";  
  }
}

In a more strict REST sense the GET mapping for the document creation is up for a debate, but in the context of this conceptual code, I can live with it ;) Also, I assume that the application gets deployed somewhere (e.g. locally) to have access to the PDF file for parsing.

Service

The service provides the actual business logic for fulfilling our requirements. It has dependencies to the (Neo4j) vector store and the (OpenAI) chat client implementation.

For processing the PDF file that can be found at the given path, it uses the PagePdfDocumentReader to represent each page of a PDF as a single Document. When adding them to the vector store by invoking vectorStore.add(parsedDocuments) it will automatically get enhanced with the embeddings generated by OpenAI before getting persisted to Neo4j.

Prompting for Cypher statements is done with a Prompt containing two parts:

A system message that defines the basis for the communication with OpenAI Chat
The actual user prompt containing the message For the first one, a prepared message with a placeholder {documents} will be used. The replacement for this will be the content of the top 4 results of a vectorStore.similaritySearch(message).

Telling the chat how to behave when a message comes in and restricting it strictly to use only the provided documents' content should only return Cypher. "Correct Cypher?" you may ask. "What it thinks correct Cypher is!" I would answer.

The whole service class looks like this:

@Service  
public class CypherPromptService {  
  
  private final VectorStore vectorStore;  
  
  private final ChatClient chatClient;  
  
  private static final String SYSTEM_PROMPT_TEMPLATE = """  
    You are an assistant that gives out Cypher code snippets.
    Use the information from the DOCUMENTS section only to provide accurate answers.
    Return just the code snippet without formatting. No descriptive text.
    Don't use any learned knowledge that is not within the DOCUMENTS section.

    DOCUMENTS:  
    {documents}""";  
  
  public CypherPromptService(VectorStore vectorStore, ChatClient chatClient) {  
   this.vectorStore = vectorStore;  
   this.chatClient = chatClient;  
  }  
    
  public String convertToCypher(String message) {  
  
   var result = vectorStore.similaritySearch(message);  
   String documents = result.stream().map(Document::getContent).collect(Collectors.joining(System.lineSeparator()));  
   var systemMessage = new SystemPromptTemplate(SYSTEM_PROMPT_TEMPLATE)  
     .createMessage(Map.of("documents", documents));  
  
   var userMessage = new UserMessage(message);  
  
   var prompt = new Prompt(List.of(systemMessage, userMessage));  
   ChatResponse response = chatClient.call(prompt);  
  
   return response.getResult().getOutput().getContent();  
  }  
  
  public void createEmbeddings(String filePath) {  
   List<Document> parsedDocuments = new PagePdfDocumentReader(filePath).get();  
   parsedDocuments.forEach(document -> document.getMetadata().clear());  
  
   vectorStore.add(parsedDocuments);  
  }  
}

Run it

We need to get the current Cypher 5 manual as a PDF. This can be found herein the docs archive. For not having to write too much, I simply put it in the main/resources folder of my application before starting the application.

The application can be started within your IDE or by calling ./mvnw spring-boot:run on the command line.

Creation of Documents with embeddings

In this curl call we need to provide the file path that the application should be looking for the PDF to create the Documents. If you followed along until here, this should be just the file name because the file was placed in the resources folder.

curl 'http://localhost:8080/create?filePath=cypher.pdf'

Don't get confused by some exceptions regarding missing fonts or even the java.lang.IllegalArgumentException: Comparison method violates its general contract! in the application logs. It will take a while until all pages of the PDF got processed and saved with their embeddings in the database.

After this call returns, the dataset has been successfully created and the actual text to Cypher conversion can be invoked.

Conversion

Now the chat-based conversion can be invoked. Let's start with something simple to see if the interaction works and we get a correct response.

curl 'http://localhost:8080/?m=Create%20a%20node'

(For convenience I am using curl but it requires the conversion of special characters, like white spaces, quotes, etc. Use a browser or something like postman for your own sanity :) )

And the response is:

CREATE (n)

This looks promising. For the next example we will use the prompt: "Create a node labelled User with a property called 'nickname' having a value 'meistermeier'".

curl 'http://localhost:8080/?m=Create%20a%20node%20labelled%20User%20with%20a%20property%20called%20%27nickname%27%20having%20a%20value%20%27meistermeier%27'

returns

CREATE (:User {nickname: 'meistermeier'})

The last bit for this example would be to create a query to get to know how many User nodes exist in the database. "How many nodes labelled 'User' exists in the database?"

curl 'http://localhost:8080/?m=%22How%20many%20nodes%20labelled%20%27User%27%20exists%20in%20the%20database?%22'

will yield

MATCH (n:User) RETURN count(n)

Conclusion

The results for expected to be easily to produce are really interesting to observe, and in this context surprised me positively. "Create a node Create a node labelled User with a property called 'nickname' having a value 'meistermeier' linked to another new node labelled User with a property called 'nickname' having a value 'rotnroll666'"

curl 'http://localhost:8080/?m=%22Create%20a%20node%20Create%20a%20node%20labelled%20User%20with%20a%20property%20called%20%27nickname%27%20having%20a%20value%20%27meistermeier%27%20linked%20to%20another%20new%20node%20labelled%20User%20with%20a%20property%20called%20%27nickname%27%20having%20a%20value%20%27rotnroll666%27%22'

Also work partially by taking some relation type from the documentation.

CREATE (a:User {nickname: 'meistermeier'})-[:FRIEND]-(b:User {nickname: 'rotnroll666'})

But this is something we have to keep in mind when using a system with a limited understanding (the PDF) and giving it a incomplete (missing relationship type) information. After presenting the prompts that will create the Cypher statements it is now up for the reader to decide if this is a valuable way to create Cypher -or even execute it blindly with the driver in the database- or if it would be better in the long run to learn and understand Cypher.

If you want to give some feedback or leave some thoughts, feel free to ping/mention me on mastodon. Any questions regarding the code? You can find the sources of this example on GitHub meistermeier/spring-ai-neo4j-example.

Cypher Language Server support for Neovim

2023-12-04T00:00:00Z

A few weeks ago, my colleagues released the cypher-language-support. Besides a vs-code plugin it contains the cypher-language-server. This means that it is possible to have Cypher support within the IDE/editor of your choice if it supports the language server protocol. The question came up in my head: How hard can it be to get this into Neovim? Disclaimer: I am a regular user of (Neo)vim but not for the projects I am working on during the (work) day.

First baby steps

To figure out what was needed to get a out-of-the-box experience in the end, I started with the installation of the cypher-language-server.

npm i -g @neo4j-cypher/language-server

And an empty Neovim configuration to see if things(tm) will work.

vim.lsp.start({
  name = 'cypher-ls',
  cmd = { 'cypher-language-server', '--stdio' },
})

Opening a file containing Cypher and attaching the LSP server manually (where 1 is the buffer and the LSP server id)

:lua vim.lsp.buf_attach_client(1, 1);

successfully shows us that the detection works and, if we create invalid content, the errors/warnings appear.

Make it convenient to use

Of course I did not want my journey end in this PoC state. My current setup started off with nvim-lua/kickstart for getting nvim-lspconfig, mason and mason-lspconfig set up in context of language server support. This means that I already have the most used plugins in place to benefit from LSP support:

nvim-lspconfig: Takes care of (default) configuration of a language server incl. the startup
mason: Management of LSP server, linter and formatter installations
mason-lspconfig: Marry the knowledge about the physical installation from mason with the nvim-lspconfig information

nvim-lspconfig

To get a working auto-configuration it was time to approach my first Neovim plugin contribution. Cloning nvim-lspconfig and making the clone the source of truth via

vim.opt.runtimepath:prepend("/<path-to-nvim-lspconfig.nvim>")

on top of my config lua, I was ready to create the needed changes.

The nvim-lspconfig plugin, as also the other ones I have seen, is really well structured. Adding the support for cypher-language-server was very straight forward. I just needed to create a new file with the name of the language server I plan to provide (cypher_ls) and added the following content.

local util = require 'lspconfig.util'

return {
  default_config = {
    cmd = { 'cypher-language-server', '--stdio' },
    filetypes = { 'cypher' },
    root_dir = util.find_git_ancestor,
    single_file_support = true,
  },
  docs = {
    description = [[
<desription content>
]],
  },
}

Et voila, with the following line I could activate "my" language server.

require('lspconfig').cypher_ls.setup{}

Now the time of truth came to figure out how the community and maintainers in Neovim(-lspconfig) land are willing to accept a random PR without previous issue or some kind of discussion upfront. To my surprise this was really not a problem at all, and after opening nvim-lspconfig#2883 just within 4 hours.

This encouraged me to go further down the Neovim plugin world, and for a really convenient experience, add support in mason and mason-lspconfig.

mason

The mason project helps to install various LSP servers, as well as linters and formatters. A potential future user of the cypher-language-server should not care about the installation process of the server but it should, if missing, get installed automatically. To make mason and its users aware of the existence of the cypher-language-server, the information has to be placed in the mason-registry (registry site). After investigating what other language server did, I was confident enough to raise a PR there mason-registry#3427. Since I did not want to fake the registry page in the mason module, it was now time to wait for me until the page got updated. Again after just a few days, the PR got merged with some changes, and the registry got an update.

At this point it was already possible to remove my manual installation of the cypher-language-server and just type

:MasonInstall cypher-language-server

to get the language server installed.

On checkmark more on my virtual list. Let's go to the next plugin.

mason-lspconfig

Mason provides the installation of the language server and nvim-lspconfig knows about how to use it. To combine those, I needed to add this bridging information to the mason-lspconfig plugin. As with the mason registry PR before, I looked for a already existing LSP server that is similar to the one I plan to add. To ensure everything works with my changes, I set up mason-lspconfig to be sourced from my local machine. Like before with nvim-lspconfig, I put this on top of my Neovim config.

vim.opt.runtimepath:prepend("/<path-to-mason-lspconfig.nvim>")

And then the moment of truth came, I uninstalled the language server once again and added cypher_ls to the mason LSP server list. Opening a cypher file, I could see that the installation process started and everything was working as expected.

Again after just a few hours, mason-lspconfig#312 got merged with some fixes and the cypher-language-server was available for every Neovim user.

File type support

Something I did not mention, but you have spotted in the nvim-lspconfig section already, is the file type the language server should work with: cypher. At the moment you have to either

:set ft=cypher after opening a file
put in // vim: set ft=cypher : anywhere in you file
or create a file ~/.config/nvim/ftdetect/cypher.vim containing

au BufNewFile,BufRead *.cypher setf cypher

Of course it would be fantastic to get also the file type support into Neovim. After a little bit of research I figured out that the addition of file types are handled by merging changes from vim to Neovim.

And here it is: my first PR for the vim project to introduce cypher as a file type: vim#13516.

After this got merged, the Neovim project took the changes and added the file type support neovim_84688ec7.

If you're not building Neovim from the sources, you have to wait for the next release to also benefit from the file type support.

Conclusion

Taken that the heavy lifting of creating a LSP server was already done by my colleagues, it was a very nice experience to contribute to their efforts and the Neovim ecosystem. I did not know upfront what or whom to expect to have interaction with. But I was positive surprised about the fast merging and the interaction in general.

If you plan to add your LSP support for those projects, I can only encourage to do so.

Maintain a simplistic blog

2023-08-07T00:00:00Z

What is nicer than just publishing blog posts? Right, figuring out how to get them published convenient.

My blog is a static build done by Jekyll which holds its post in the _posts folder. The articles themselves are written in Obsidian.md.

Frederik asked me about integrations or automations...

... I didn't know about any that suits my idea.

Copy blog posts (Single source of truth)

We cannot directly copy the posts because the files are named by their title. Jekyll uses Front Matter to get some metadata into the posts. Luckily for me, also Obsidian supports this feature. In the beginning of a note, I can add for example

---
layout: post
file_name: blog_infrastructure
title: "Maintain a simplistic blog"
description: "Hacking around to be lazy later"
date: 2023-08-07
categories: git obsidian ios
---

I want to follow some basic steps here

Take everything in the Blog folder that has a md file extension
Copy and rename the file to (my) Jekyll pattern YEAR-MONTH-DAY-shortend-title.md

Scripting

And now the fun already begins. You can skip this section and directly jump to the bash block in the end if you are not interested in my thoughts during the process. To have a loose coupling between the physical file and the blog post, I have to extract the date from the metadata. For this I use yq.

cat Maintain\ a\ simplistic\ blog.md| yq --front-matter=extract ".date"
2023-07-10

Now let's head over to the second part of the file name. Shorting it down to a fixed size would be enough, also replacing whitespaces with underscores (_) will be fine. Because I expect the input to come from "somewhere" (currently find), I am aware that I will get the whole path of the file in the first place. After playing with cut and awk I realized that it might be much easier and controllable to add the file_name to the FrontMatter metadata.

Images

Obsidian is pretty much lenient, when it comes to image references, but of course not as lenient as I thought. Having images in a separate folder, like the Jekyll standard structure suggests, allows to reference the images either just by name (myimage.png), or (partial) path (media/blogpost/myimage.png). As a consequence the rendered page expects the images to be either in the same folder domain.xyz/2023/08/15/myimage.png or as a sub-path domain.xyz/2023/08/15/media/blogpost/myimage.png. After trying to figure out if there is any built-in way to a) see the correct images in Obsidian and b) also getting the references rendered in the Jekyll site build, I decided to use a custom plugin https://github.com/nhoizey/jekyll-postfiles and bring them closer to the posts. For now, I cannot imagine getting problems by using this structure.

group :jekyll_plugins do
  gem "jekyll-feed", "~> 0.12"
  gem "jekyll-postfiles"
end

The only change will be now to create a folder for each post. In the end the script gets even a little bit shorter.

#!/usr/bin/env bash

export TARGET=<path/to/Jekyll/_posts>

cp -r * "$TARGET"
find "$TARGET" -name "*.md" -exec sh -c '
DATE_PART=$(cat "$1" |yq --front-matter=extract ".date");
FILE_NAME=$DATE_PART"-"$(cat "$1" | yq --front-matter=extract ".file_name");
DIRECTORY=$(dirname "$1")
mv "$1" "$DIRECTORY/$FILE_NAME.md"
' unused {} \;

Folder structure

The resulting folder structure in Obsidian looks like this. Not that it could have as many sub-folders as you want. The folder names are for pretending to be organized ;)

├── duckdb
│   └── First experiences with DuckDB (and SQL in general after a while).md
├── neo4j
│   └── Spring Data Neo4j GraphQL 1
│       ├── Spring Data Neo4j GraphQL 1.md
│       ├── example_flow.png
│       ├── graph_data_set.png
│       └── graphiql.png
└── random
    └── Maintain a simplistic blog
        └── Maintain a simplistic blog.md

Jekyll configuration

In the _posts folder used to be also the archived posts migrated from the my old blog. To have this folder exclusively available for the new posts imported from Obsidian, I created a new _archive folder and added is as a collection to my config.yml.

collections:
  archive:
    output: true

Every post in there is part of the Jekyll repository while the new posts are only in place during the build steps. Mainly because I don't care enough to import them into Obsidian.

Automation

Because it is available to me, I use a friend's GitLab setup. I am pretty sure that everything written here can be done with GitHub Actions, Codeberg Actions or the CI of your choice.

Blog sources repository

I tried various ways to not do this, but in the end, I needed to put (parts of) my Obsidian vault (which is shared via iCloud) into a git repository. After adding the script to the CI file, I ended up with:

package:
  image: alpine
  script: 
    - apk add yq
    - |-
        find ./ -name "*.md" -exec sh -c '
        DATE_PART=$(cat "$1" |yq --front-matter=extract ".date");
        FILE_NAME=$DATE_PART"-"$(cat "$1" | yq --front-matter=extract ".file_name");
        DIRECTORY=$(dirname "$1")
        mv "$1" "$DIRECTORY/$FILE_NAME.md"
        ' unused {} \;
  artifacts:
        paths:
            - ./*
        exclude:
          - ".git/**/*"
          - ".git"
          - ".git/"
          - ".gitlab-ci.yml"
deploy:
  trigger: meistermeier/Blog

A small alpine Docker image with just yq added to the mix, my script that can be boiled down to working in the current directory, and some exclusions to not push the whole git and config into the world later. After the packaging went through, it triggers the Jekyll repository to do it's job.

The Jekyll repository

Here the journey of the blog posts continues. Since a pipeline script says more than thousand words, here it is:

stages:
  - build
  - upload

generate:
  stage: build
  image: jekyll/minimal
  script:
    - apk update
    - apk upgrade
    - apk add curl
    - 'curl --location --output artifacts.zip --header "PRIVATE-TOKEN: $ACCESS_TOKEN" "https://gitlab.server/api/v4/projects/meistermeier%2FObsidian-Sources/jobs/artifacts/main/download?job=package"'
    - unzip artifacts.zip -d _posts
    - bundle install
    - jekyll build
  artifacts:
    expire_in: 1 hours
    paths:
      - _site/
upload:
  stage: upload
  image: alpine
  script: 
    - apk add lftp
    - lftp -c "<send things over the wire>"

I have two stages (mostly to skip the upload part while experimenting with the pipeline): build and upload. The build stage fetches the latest artifacts archive from the upstream project containing the blog posts and puts it into the target Jekyll structure (_posts). To get the build support for the custom plugin, I mentioned above, it is necessary to invoke bundle install to have it in place when Jekyll needs it during processing. After this, the build command gets invoked and produces the site. After this, the site will get deployed (here by using lftp) in the upload stage to the web host.

Conclusion

This took by far a lot longer to get "right" than I have expected. In the end I hope to give you some hints on how you could also create a pipeline without hitting every trap on the way, as I did. At least for me this works as I wanted it do. Also, the two projects clearly show the separation of concerns. There is now the content repository with pure Obsidian markdown (and images) and the infrastructure repository that, besides the archived posts, contains only the Jekyll page itself.

Happy blogging.

First experiences with DuckDB (and SQL in general after a while)

2023-07-06T00:00:00Z

Disclaimer

I never did hardcore SQL in my career. If you are a (PL/)SQL developer please be kind while judging me for the following content ;)

Setup

Install the DuckDB command line https://duckdb.org/docs/installation/index For the example data, I have taken a CSV export from the Garmin activities https://connect.garmin.com/modern/activities I took this one and not GPX/TPX files because I did not want to jump directly into spatial or deeper analytics during my first steps. In case you don't have data stored on the Garmin page, you can take the content from the end of this post and store it in a file called Activities.csv. If you wonder why the CSV just shows a few entries: You need to scroll to the end of the page or at least so far that you get all the data you want to be in the CSV, what an ugly user experience.

Hello DuckDB

In the folder with the CSV file, call duckdb to start be ready to go. First of all let's have a look at the structure of the CSV file.

DESCRIBE SELECT * FROM 'Activities.csv';

will give us an overview of the auto-detected types within the CSV file:

column_name	column_type	null
Activity Type	VARCHAR	YES
Date	TIMESTAMP	YES
Favorite	BOOLEAN	YES
Title	VARCHAR	YES
Distance	VARCHAR	YES
Calories	VARCHAR	YES
Time	TIME	YES
Avg HR	BIGINT	YES
Max HR	BIGINT	YES
Aerobic TE	DOUBLE	YES
Avg Cadence	VARCHAR	YES
Max Cadence	VARCHAR	YES
Avg Pace	VARCHAR	YES
Best Pace	VARCHAR	YES
Total Ascent	VARCHAR	YES
Total Descent	VARCHAR	YES
Avg Stride Length	DOUBLE	YES
Avg Vertical Ratio	DOUBLE	YES
Avg Vertical Oscillation	DOUBLE	YES
Avg Ground Contact Time	BIGINT	YES
Training Stress Score®	DOUBLE	YES
Avg Power	BIGINT	YES
Max Power	BIGINT	YES
Grit	DOUBLE	YES
Flow	DOUBLE	YES
Total Strokes	VARCHAR	YES
Avg. Swolf	BIGINT	YES
Avg Stroke Rate	BIGINT	YES
Total Reps	BIGINT	YES
Dive Time	VARCHAR	YES
Min Temp	DOUBLE	YES
Surface Interval	VARCHAR	YES
Decompression	VARCHAR	YES
Best Lap Time	VARCHAR	YES
Number of Laps	BIGINT	YES
Max Temp	DOUBLE	YES
Avg Resp	VARCHAR	YES
Min Resp	VARCHAR	YES
Max Resp	VARCHAR	YES
Moving Time	TIME	YES
Elapsed Time	TIME	YES
Min Elevation	VARCHAR	YES
Max Elevation	VARCHAR	YES

Looking at the result, the auto-detection of types does a quite good job. But it is easy to spot that e.g. the elevation columns fall back into a varchar type instead being numeric. This is a problem that will get solved later.

Some information that is missing for a proper table entries is an identifier. To work later with references for joins etc. we should take this CSV file as the source for a new table with an id based on the uuid() function for each entry to get this additional field.

CREATE TABLE activity_import AS SELECT uuid() AS id, * FROM 'Activities.csv';

Activity table

Since not every activity has all values present, the data will get imported in multiple tables. The first one would be some kind of base data table, where Activity Type, Date and Title get stored. To derive the new table from the import table, the statement

CREATE TABLE activity AS SELECT id, "Activity Type", "Date", "Title" FROM activity_import;

Verifying that the expected data is in the table with

SELECT * FROM ACTIVITY LIMIT 10;

produces

id	Activity Type	Date	Title
14cd31a9-868b-4d3d-a507-373850d90ac7	Walking	2023-07-04 08:01:08	Brunswick Walking
2e5480b4-4b2f-4dea-a340-515b04fa2083	Walking	2023-07-02 08:18:50	Brunswick Walking
19663dd5-d105-4471-91e6-69e85400ad5f	Walking	2023-06-30 15:54:38	Brunswick Walking
596c3c8d-ae86-4953-bd7f-258d23ea2726	Running	2023-06-28 07:52:00	Brunswick Running
e71833bb-f4c3-4e4e-ac02-6d183d202e0d	Running	2023-06-26 07:52:45	Brunswick Running
419acad8-2ca4-48a1-8f1a-9fc7c4263918	Running	2023-06-22 07:53:57	Brunswick Running
a6f9a970-80f4-4bc0-ad95-3e38a7b5c3e4	Pool Swimming	2023-06-17 11:39:30	Pool Swimming
7c687f12-492f-49d2-9647-68c063816e4f	Running	2023-05-28 12:20:50	Heiligenhafen Running
b4ae30e8-d498-4448-866a-184b3f8245b4	Running	2023-05-27 07:49:10	Heiligenhafen Running
faedde1e-295d-4566-9d09-1358a9adfa1a	Walking	2023-04-14 15:53:40	Brunswick Walking

Elevation Comparison and Type conversion

Now let's take on the problem with the varchar for the elevation columns that we saw in the beginning. To figure out which values are non-numeric in the Min Elevation column, run:

SELECT "Min Elevation" FROM activity_import WHERE try_cast("Min Elevation" AS NUMERIC) IS NULL ORDER BY "Min Elevation";

That will uncover the "faulty" entries.

Min Elevation
--
--
--
--
--
--
--
1,246
1,246
1,269
1,312
1,318
1,321
1,353
1,412
1,448
1,449
1,449
1,458
1,475
1,476
1,476
1,476
1,476
1,476
1,477
1,478
1,478
1,478
1,478
1,478
1,483
1,494
1,524

There seem to be some empty fields indicated with -- and the thousands marker cannot be parsed. At least a pattern now to filter and alter the values before we import them into a table.

To convert the column into a numeric type, the pre-processing is done this way case when "Min Elevation"=='--' THEN 0 ELSE replace("Min Elevation", ',', '')::numeric END. Now a table activities_elevation can be created combining the statement above with the CREATE statement. Also, the conversion for the Max Elevation will get added to the new table because it is the very same as for the Min Elevation.

CREATE TABLE activity_elevation AS SELECT id,
  case when "Min Elevation"=='--' THEN 0 ELSE replace("Min Elevation", ',', '')::numeric END as "Min Elevation",
  case when "Max Elevation"=='--' THEN 0 ELSE replace("Max Elevation", ',', '')::numeric END as "Max Elevation"
  FROM activity_import;

Querying

The question that I asked myself first after having a glance at the data was: "Which activity had the most span between min and max elevation". Of course in the real world this does not cover those activities with a lot of smaller ascents that repeat over the track. The query

SELECT "Activity Type", "Date", "Max Elevation", "Min Elevation", "Max Elevation" - "Min Elevation" AS difference 
	FROM activity_elevation 
	JOIN activity USING (id) 
ORDER BY difference DESC
LIMIT 10;

returns

Activity Type	Date	Max Elevation	Min Elevation	difference
Hiking	2022-08-01 11:02:30	2356.000	1353.000	1003.000
Cycling	2020-08-09 09:47:40	1924.000	1269.000	655.000
Cycling	2022-08-05 13:23:41	1953.000	1412.000	541.000
Cycling	2022-08-05 10:04:12	1684.000	1246.000	438.000
Walking	2021-08-05 11:13:26	1361.000	938.000	423.000
Hiking	2020-08-15 14:58:34	1940.000	1524.000	416.000
Cycling	2020-09-04 09:24:56	538.000	137.000	401.000
Hiking	2022-08-09 09:23:26	1783.000	1477.000	306.000
Hiking	2022-08-07 09:35:59	1731.000	1448.000	283.000
Cycling	2021-08-11 09:40:48	1524.000	1246.000	278.000

Those are my Top10 activities regarding elevation. And what surprises me the most: The join with the directive USING (id) just works as expected by matching the id columns of both tables.

Table-less

You might want to do everything in-place and just use DuckDB as a fancy CSV tool. The whole steps above can also be combined into a single statement:

SELECT "Activity Type", "Date",
	case when "Min Elevation"=='--' THEN 0 ELSE replace("Min Elevation", ',', '')::numeric END as "MinElevation",
	case when "Max Elevation"=='--' THEN 0 ELSE replace("Max Elevation", ',', '')::numeric END as "MaxElevation",
	"MaxElevation" - "MinElevation" AS difference
	from 'Activities.csv'
ORDER BY difference DESC 
LIMIT 10;

And here is the statement that I have used to create the example CSV file from my original export.

COPY (SELECT "Activity Type", "Min Elevation", "Max Elevation",
	concat("Activity Type", ' on ', times.Date::varchar) as Title,
	times.Date as Date
	from 'Activities_original.csv' TABLESAMPLE 20 POSITIONAL JOIN 
		(SELECT generate_series as Date from generate_series(TIMESTAMP '2023-04-10 08:00:00', TIMESTAMP '2023-06-11', INTERVAL 36 HOUR) LIMIT 20) as times
	)
TO 'Activities.csv' WITH (HEADER 1);

Activities.csv

Activity Type,Min Elevation,Max Elevation,Title,Date
Running,75,102,Running on 2023-04-10 08:00:00,2023-04-10 08:00:00
Running,"1,478","1,547",Running on 2023-04-11 20:00:00,2023-04-11 20:00:00
Cycling,65,94,Cycling on 2023-04-13 08:00:00,2023-04-13 08:00:00
Cycling,"1,246","1,684",Cycling on 2023-04-14 20:00:00,2023-04-14 20:00:00
Running,74,107,Running on 2023-04-16 08:00:00,2023-04-16 08:00:00
Hiking,"1,483","1,721",Hiking on 2023-04-17 20:00:00,2023-04-17 20:00:00
Running,-8,8,Running on 2023-04-19 08:00:00,2023-04-19 08:00:00
Walking,87,96,Walking on 2023-04-20 20:00:00,2023-04-20 20:00:00
Running,74,107,Running on 2023-04-22 08:00:00,2023-04-22 08:00:00
Running,74,107,Running on 2023-04-23 20:00:00,2023-04-23 20:00:00
Running,74,107,Running on 2023-04-25 08:00:00,2023-04-25 08:00:00
Running,75,102,Running on 2023-04-26 20:00:00,2023-04-26 20:00:00
Running,70,107,Running on 2023-04-28 08:00:00,2023-04-28 08:00:00
Gravel/Unpaved Cycling,76,93,Gravel/Unpaved Cycling on 2023-04-29 20:00:00,2023-04-29 20:00:00
Pool Swimming,--,--,Pool Swimming on 2023-05-01 08:00:00,2023-05-01 08:00:00
Cycling,137,538,Cycling on 2023-05-02 20:00:00,2023-05-02 20:00:00
Walking,964,"1,085",Walking on 2023-05-04 08:00:00,2023-05-04 08:00:00
Walking,75,102,Walking on 2023-05-05 20:00:00,2023-05-05 20:00:00
Cycling,77,112,Cycling on 2023-05-07 08:00:00,2023-05-07 08:00:00
Running,"1,449","1,545",Running on 2023-05-08 20:00:00,2023-05-08 20:00:00

Using Spring for GraphQL With Spring Data Neo4j

2023-06-22T00:00:00Z

Introduction

A few weeks ago version 1.2.0 of Spring (for) GraphQL was released with a bunch of new features. This also includes even better integration with Spring Data modules. Motivated by those changes, more support in Spring Data Neo4j has been added, to give the best experience when using it in combination with Spring GraphQL. This post will guide you on creating a Spring application with data stored in Neo4j and GraphQL support. If you are only partial interested in the domain, you can happily skip the next section ;)

Domain

For this example I have chosen to reach into the Fediverse. More concrete, to put some Servers and User into the focus. Why the domain was picked up for this, is now for the reader to discover in the following paragraphs.

The data itself is aligned to the properties that could be fetched from the Mastodon API. To keep the data set simple the data was created by hand instead of fetching everything. This results into an easier to inspect data set. The Cypher import statements looks like this:

CREATE (s1:Server {
    uri:'mastodon.social', title:'Mastodon', registrations:true,
    short_description:'The original server operated by the Mastodon gGmbH non-profit'})
CREATE (meistermeier:Account {id:'106403780371229004', username:'meistermeier', display_name:'Gerrit Meier'})
CREATE (rotnroll666:Account {id:'109258442039743198', username:'rotnroll666', display_name:'Michael Simons'})
CREATE
(meistermeier)-[:REGISTERED_ON]->(s1),
(rotnroll666)-[:REGISTERED_ON]->(s1)

CREATE (s2:Server {
    uri:'chaos.social', title:'chaos.social', registrations:false,
    short_description:'chaos.social – a Fediverse instance for & by the Chaos community'})
CREATE (odrotbohm:Account {id:'108194553063501090', username:'odrotbohm', display_name:'Oliver Drotbohm'})

CREATE
(odrotbohm)-[:REGISTERED_ON]->(s2)

CREATE
(odrotbohm)-[:FOLLOWS]->(rotnroll666),
(odrotbohm)-[:FOLLOWS]->(meistermeier),
(meistermeier)-[:FOLLOWS]->(rotnroll666),
(meistermeier)-[:FOLLOWS]->(odrotbohm),
(rotnroll666)-[:FOLLOWS]->(meistermeier),
(rotnroll666)-[:FOLLOWS]->(odrotbohm)

CREATE
(s1)-[:CONNECTED_TO]->(s2)

After running the statement, the graph forms this shape.

Noticeable information is that even all users follow each others, the Mastodon servers are only connected in one direction. Users on server chaos.social cannot search or explore timelines on mastodon.social.

Disclaimer: The federation of the servers is made up with a non-bidirectional relationship for this example.

Components

To follow along with the example shown, you should use the following minimum versions:

Spring Boot 3.1.1 (which includes the following)
- Spring Data Neo4j 7.1.1
- Spring GraphQL 1.2.1
Neo4j version 5

Best is to head over to https://start.spring.io and create a new project with Spring Data Neo4j and Spring GraphQL dependency. If you are a little bit lazy, you could also download the empty project from this link.

To follow along the example 100%, you would need to have Docker installed on your system. If you don’t have this option or don’t want to use Docker, you could use either Neo4j Desktop or the plain Neo4j Server artifact for local deployment, or as hosted options Neo4j Aura or an empty Neo4j Sandbox. There will be a note later how to connect to a manual started instance. The use of the enterprise edition is not necessary and everything works with the community edition.

First Spring for GraphQL steps

In this example the heavy lifting of configuration will be done by the Spring Boot autoconfiguration. There is no need to set up the beans manually. To find out more about what happens behind the scenes, please have a look at the Spring for GraphQL documentation. Later, specific sections of the documentation will be referenced.

Entity and Spring Data Neo4j setup

First thing to do is to model the domain classes. As already seen in the import, there are only Servers and Accounts.

@Node
public class Account {
    @Id
    private final String id; // (1)
    private final String username;
    @Property("display_name")
    private final String displayName; // (2)
    @Relationship("REGISTERED_ON")
    private final Server server;
    @Relationship("FOLLOWS")
    private List<Account> following;
    // constructor, etc.
}

(1) Given Mastodon’s id strategy for user ids, it is valid to assume, that the id is (server) unique.

(2) Here and a few lines below in the Server, @Property is used to map the database field display_name to camel-case displayName in the Java entity.

@Node
public class Server {
    @Id
    private final String uri;
    private final String title;
    @Property("registrations")
    private final Boolean registrationsAllowed;
    @Property("short_description")
    private final String shortDescription;
    @Relationship("CONNECTED_TO")
    private List<Server> connectedServers;
    // constructor, etc.
}

With these entity classes, a AccountRepository can be created.

@GraphQlRepository // (1)
public interface AccountRepository extends Neo4jRepository<Account, String>
{
}

(1) Details why this annotation is used will follow later. Here for completeness of the interface.

To connect to the Neo4j instance, the connection parameters needs to be added to the application.properties file.

spring.neo4j.uri=neo4j://localhost:7687
spring.neo4j.authentication.username=neo4j
spring.neo4j.authentication.password=verysecret

If not happened yet, the database can be started and the Cypher statement from above run to set up the data. In a later part of this article, Neo4j-Migrations will get used to be sure that the database is always in the desired state.

Spring for GraphQL setup

Before looking into the integration features of Spring Data and Spring for GraphQL, the application will get set up with a @Controller stereotype annotated class. The controller will get registered by Spring for GraphQL as a DataFetcher for the query accounts.

@Controller
public class AccountController {

    private final AccountRepository repository;

    @Autowired
    public AccountController(AccountRepository repository) {
            this.repository = repository;
    }

    @QueryMapping
    public List<Account> accounts() {
            return repository.findAll();
    }
}

Defining a GraphQL schema, that defines not only our entities but also the query with the same name as the method in the controller (accounts).

type Query {
    accounts: [Account]!
}
type Account {
    id: ID!
    username: String!
    displayName: String!
    server: Server!
    following: [Account]
    lastMessage: String!
}
type Server {
    uri: ID!
    title: String!
    shortDescription: String!
    connectedServers: [Server]
}

Also, to browse the GraphQL data in an easy way, GraphiQL should be enabled in the application.properties. This is a helpful tool during development time. Usually this should be disabled for production deployment.

spring.graphql.graphiql.enabled=true

First run

If everything is set up as described above, the application can be started with ./mvnw spring-boot:run. Browsing to http://localhost:8080/graphiql?path=/graphql will present the GraphiQL explorer.

To verify that the accounts method is working, a GraphQL request is sent to the application.

{
    accounts {
        username
    }
}

and the expected answer gets returned from the server.

{
    "data": {
    "accounts": [
        {
        "username": "meistermeier"
        },
        {
        "username": "rotnroll666"
        },
        {
        "username": "odrotbohm"
        }
    ]
    }
}

Of course the method in the controller can be tweaked by adding parameters for respecting arguments with @Argument or getting the requested fields (here accounts.username) to squeeze down the amount of data that gets transported over the network. In the previous example, the repository will fetch all properties for the given domain entity, including all relationships. This data will get mostly discarded to return only the username to the user.

This example should give an impression of what can be done with Annotated Controllers. Adding the query generation and mapping capabilities of Spring Data Neo4j a (simple) GraphQL application was created.

But at this point both libraries seem to live in parallel in this application and not yet like an integration. How can SDN and Spring for GraphQL get really combined?

Spring Data Neo4j GraphQL integration

As a first step, the accounts method from the AccountController can be deleted. Restarting the application and querying it again with the request from above will still bring up the same result.

This works because Spring for GraphQL recognizes the result type (array of) Account from the GraphQL schema. It scans for eligible Spring Data repositories that matches the type. Those repositories have to extend the QueryByExampleExecutor or QuerydslPredicateExecutor (not part of this blog post) for the given type. In this example, the AccountRepository is already implicitly marked as QueryByExampleExecutor because it is extending the Neo4jRespository, that is already defining the executor. The @GraphQlRepository annotation makes Spring for GraphQL aware that this repository can and should be used for the queries, if possible.

Without any changes to the actual code, a second Query field can be defined in the schema. This time it should filter the results by username. A username looks unique at the first glance but in the Fediverse this is only true for a given instance. Multiple instances could have the very same usernames in place. To respect this behaviour, the query should be able to return an array of Accounts.

The documentation about query by example (Spring Data commons) provides more details about the inner workings of this mechanism.

type Query {
    account(username: String!): [Account]!
}

Restarting the app will now present the option to add a username interactively as a parameter to the query.

{
    account(username: "meistermeier") {
    username
    following {
        username
        server {
         uri
        }
    }
    }
}

Obviously, there is only one Account with this username.

{
    "data": {
    "account": [
        {
        "username": "meistermeier",
        "following": [
            {
            "username": "rotnroll666",
            "server": {
                "uri": "mastodon.social"
            }
            },
            {
            "username": "odrotbohm",
            "server": {
                "uri": "chaos.social"
            }
            }
        ]
        }
    ]
    }
}

Behind the scenes Spring for GraphQL adds the field as a parameter to the object that gets passed to the repositories as an example. Spring Data Neo4j then inspects the example and creates matching conditions for the Cypher query, executes it and sends the result back to Spring GraphQL for further processing to shape the result into the right response format.

Pagination

Although the example data set is not this huge, it’s often useful to have a proper functionality in place that allows to request the resulting data in chunks. Spring for GraphQL uses the Cursor Connections specification.

A complete schema specification with all types looks like this.

type Query {
    accountScroll(username:String, first: Int, after: String, last: Int, before:String): AccountConnection
}

type Account {
    id: ID!
    username: String!
    displayName: String!
    server: Server!
    following: [Account]
    lastMessage: String!
}

type Server {
    uri: ID!
    title: String!
    shortDescription: String!
    connectedServers: [Server]
}

type AccountConnection {
    edges: [AccountEdge]!
    pageInfo: PageInfo!
}

type AccountEdge {
    node: Account!
    cursor: String!
}

type PageInfo {
    hasPreviousPage: Boolean!
    hasNextPage: Boolean!
    startCursor: String
    endCursor: String
}

Even though I personally like to have a complete valid schema, it is possible to skip all the Cursor Connections specific parts in the definition. Just the query with the AccountConnection definition is sufficient for Spring for GraphQL to derive and fill in the missing bits. The parameters read as following

first: the amount of data to fetch if there is no default
after: scroll position after the data should be fetched
last: the amount of data to fetch before the before position
before: scroll position until (exclusive) the data should be fetched

One question remains: In which order is the result set returned? A stable sort order is a must in this scenario, otherwise there is no guarantee that the database returns the data in a predictable order. The repository needs also to extend the QueryByExampleDataFetcher.QueryByExampleBuilderCustomizer and implement the customize method. In there it is also possible to add the default limit for the query, in this case 1 to show the pagination in action.

Added sort ordering (and limit)

@GraphQlRepository
public interface AccountRepository extends Neo4jRepository<Account, String>,
        QueryByExampleDataFetcher.QueryByExampleBuilderCustomizer<Account>
{
  @Override
  default QueryByExampleDataFetcher.Builder<Account, ?> customize(QueryByExampleDataFetcher.Builder<Account, ?> builder) {
    return builder.sortBy(Sort.by("username"))
      .defaultScrollSubrange(new ScrollSubrange(ScrollPosition.offset(), 1, true));
    }
}

After the application has restarted, it is now possible to call the first pagination query.

    {
      accountScroll {
        edges {
          node {
            username
          }
        }
        pageInfo {
          hasNextPage
          endCursor
        }
      }
    }

To get also the metadata for further interaction, some parts of the pageInfo got also requested.

{
"data": {
    "accountScroll": {
    "edges": [
        {
        "node": {
            "username": "meistermeier"
        }
        }
    ],
    "pageInfo": {
        "hasNextPage": true,
        "endCursor": "T18x"
    }
    }
}
}

Now the endCursor can be used for the next interaction. Querying the application with this as the value for after and with a limit of 2…

{
    accountScroll(after:"T18x", first: 2) {
    edges {
        node {
        username
        }
    }
    pageInfo {
        hasNextPage
        endCursor
    }
    }
}

…results in the last element(s). Also, the marker that there is no next page (hasNextPage=false) indicates that the pagination reached the end of the data set.

{
    "data": {
    "accountScroll": {
        "edges": [
        {
            "node": {
            "username": "odrotbohm"
            }
        },
        {
            "node": {
            "username": "rotnroll666"
            }
        }
        ],
        "pageInfo": {
        "hasNextPage": false,
        "endCursor": "T18z"
        }
    }
    }
}

It is also possible to scroll through the data backwards by using the defined last and before parameters. Also, it is completely valid to combine this scrolling with the already known features of query by example and define a query in the GraphQL schema that also accepts fields of the Account as filter criteria.

accountScroll(username:String, first: Int, after: String, last: Int, before:String): AccountConnection

Let’s federate

One of the big advantages in using GraphQL is the option to introduce federated data. In a nutshell this means that data stored, e.g. in the database of the application, can be enriched, like in this case, with data from a remote system / microservice / <you name it>. In the end, the data will get presented via the GraphQL surface as one entity. The consumer should not need to care about that multiple systems assembled this result.

This data federation can be implemented by making use of the already defined controller.

@Controller
public class AccountController {

    @SchemaMapping
    public String lastMessage(Account account) {
        var id = account.getId();
        String serverUri = account.getServer().getUri();

        WebClient webClient = WebClient.builder()
                        .baseUrl("https://" + serverUri)
                        .build();

        return webClient.get()
                        .uri("/api/v1/accounts/{id}/statuses?limit=1", id)
                        .exchangeToMono(clientResponse ->
                            clientResponse.statusCode().equals(HttpStatus.OK)
                            ? clientResponse
                                    .bodyToMono(String.class)
                                    .map(AccountController::extractData)
                            : Mono.just("could not retrieve last status")
                        )
                        .block();
    }

}

Adding the field lastMessage to the Account in the schema and restarting the application, gives now the option to query for the accounts with this additional information.

{
  accounts {
	username
	lastMessage
  }
}

{
    "data": {
    "accounts": [
        {
        "username": "meistermeier",
        "lastMessage": "@taseroth erst einmal schauen, ob auf die Aussage auch Taten folgen ;)"
        },
        {
        "username": "odrotbohm",
        "lastMessage": "Some #jMoleculesp/#SpringCLI integration cooking to easily add the former[...]"
        },
        {
        "username": "rotnroll666",
        "lastMessage": "Werd aber das Rad im Rückwärts-Turbo schon irgendwie vermissen."
        }
    ]
    }
}

Looking at the controller again, it becomes clear that the retrieval of the data is quite a bottleneck right now. For every Account a request gets issued after another. But Spring for GraphQL helps to improve the situation of the ordered requests for each Account after another. The solution is to use @BatchMapping on the lastMessage field in contrast to @SchemaMapping.

@Controller
public class AccountController {
  @BatchMapping
  public Flux<String> lastMessage(List<Account> accounts) {
    return Flux.concat(
      accounts.stream().map(account -> {
        var id = account.getId();
        String serverUri = account.getServer().getUri();
        WebClient webClient = WebClient.builder()
                        .baseUrl("https://" + serverUri)
                        .build();

        return webClient.get()
                        .uri("/api/v1/accounts/{id}/statuses?limit=1", id)
                        .exchangeToMono(clientResponse ->
                          clientResponse.statusCode().equals(HttpStatus.OK)
                          ? clientResponse
                                  .bodyToMono(String.class)
                                  .map(AccountController::extractData)
                          : Mono.just("could not retrieve last status")
                        );
    }).toList());
  }
}

To improve this situation even more, it is recommended to also introduce proper caching to the result. It might not be necessary that the federated data gets fetched on every request but only refreshed after a certain period.

Testing and test data

Neo4j-Migrations

Neo4j-Migrations is a project that applies migrations to Neo4j. To be sure that always a clean state of the data is present in the database, an initial Cypher statement is provided. It has the same content as the Cypher snippet in the beginning of this post. In fact, the content is included directly from this file.

Putting Neo4j-Migrations on the classpath by providing the Spring Boot starter, it will run all migrations from the default folder (resources/neo4j/migrations).

<dependency>
    <groupId>eu.michael-simons.neo4j</groupId>
    <artifactId>neo4j-migrations-spring-boot-starter</artifactId>
    <version>${neo4j-migrations.version}</version>
    <scope>test</scope>
</dependency>

Testcontainers

Spring Boot 3.1 comes with a new features for Testcontainers. One of this feature is the automatic setting of properties without the need to define @DynamicPropertySource. The (to Spring Boot known) properties will get populated at test execution time after the container has started.

First the dependency definition for Testcontainers Neo4j is needed in our pom.xml.

<dependency>
    <groupId>org.testcontainers</groupId>
    <artifactId>neo4j</artifactId>
    <scope>test</scope>
</dependency>

<dependency>
    <groupId>org.testcontainers</groupId>
    <artifactId>junit-jupiter</artifactId>
    <scope>test</scope>
</dependency>

To make use of Testcontainers Neo4j, a container definition interface will be created.

public interface Neo4jContainerConfiguration {

    @Container
    @ServiceConnection
    Neo4jContainer<?> neo4jContainer = new Neo4jContainer<>(DockerImageName.parse("neo4j:5"))
            .withRandomPassword()
            .withReuse(true);

}

This can then be used with the @ImportTestContainers annotation in the (integration) test class.

@SpringBootTest
@ImportTestcontainers(Neo4jContainerConfiguration.class)
public class Neo4jGraphqlApplicationTests {

    private final GraphQlTester graphQlTester;

    @Autowired
    public Neo4jGraphqlApplicationTests(ExecutionGraphQlService graphQlService) {
        this.graphQlTester = ExecutionGraphQlServiceTester.builder(graphQlService).build();
    }

    @Test
    void resultMatchesExpectation() {
        String query = "{" +
                "  account(username:\"meistermeier\") {" +
                "    displayName" +
                "  }" +
                "}";

        this.graphQlTester.document(query)
                .execute()
                .path("account")
                .matchesJson("[{\"displayName\":\"Gerrit Meier\"}]");

    }

}

For completeness this test class also includes the GraphQlTester and an example how to test the application’s GraphQL API.

Testcontainers at development time

It is also now possible to run the whole application directly from the test folder and use a Testcontainers image.

@TestConfiguration(proxyBeanMethods = false)
public class TestNeo4jGraphqlApplication {

        public static void main(String[] args) {
                SpringApplication.from(Neo4jGraphqlApplication::main)
                                .with(TestNeo4jGraphqlApplication.class)
                                .run(args);
        }

        @Bean
        @ServiceConnection
        public Neo4jContainer<?> neo4jContainer() {
                return new Neo4jContainer<>("neo4j:5").withRandomPassword();
        }

}

The @ServiceConnection annotation also takes care that the application started from the test class knows the coordinates the container is running at (connection string, username, password…).

To start the application outside the IDE, it is now also possible to invoke ./mvnw spring-boot:test-run. If there is only one class with a main method in the test folder, it will get started.

Topics left out / Try it

In parallel to the QueryByExampleExecutor, support for QuerydslPredicateExecutor exists in the Spring Data Neo4j module. To make use of it, the repository needs to extend the CrudRepository instead of the Neo4jRepository and also declare it as a QuerydslPredicateExecutor for the given type. Adding support for scrolling/pagination would require to also add the QuerydslDataFetcher.QuerydslBuilderCustomizer and implement its customize method.

The whole infrastructure presented in this blog post is also available for the reactive stack. Basically prefixing everything with Reactive... (like ReactiveQuerybyExampleExecutor) will turn this into a reactive application.

Last but not least, the scroll mechanism used here is based on an OffsetScrollPosition. There is also a KeysetScrollPosition that can be used. It makes use of the sort property/properties in combination with the defined id.

@Override
default QueryByExampleDataFetcher.Builder<Account, ?> customize(QueryByExampleDataFetcher.Builder<Account, ?> builder) {
        return builder.sortBy(Sort.by("username"))
                        .defaultScrollSubrange(new ScrollSubrange(ScrollPosition.keyset(), 1, true));
}

Summary

It’s nice to see how convenient methods in Spring Data modules do not only provide a broader accessibility for users' use-cases, but also get used by other Spring projects to reduce the amount of code that needs to be written. This results in less maintenance for the existing code base and helps focussing on the business problem instead of infrastructure.

This post got a little bit longer because I explicitly want to touch at least the surface on what is happening when a query gets invoked without speaking just about the automagical result.

Please go ahead and explore more about what is possible and how the application behaves for different types of queries. It is near to impossible to cover every topic and feature that is available within one blog post.

Happy GraphQL coding and exploring.

Gerrit's Blog

How to create a simplistic blog for your terminal

Parse pages for the blog

Using yq for metadata extraction

Render the blog post

The colored index page

Short ANSI color command detour

Using colors

Automate and build

Serving the curl clients

Spring Actuator Security

Setup

Enabling the heapdump endpoint

Adding some credentials

Heapdump analysis

Bytes and obfuscation

Externalizing the credentials

Securing the endpoint

Enable Cypher Language Server for Neovim

Requirements

Setting up the needed infrastructure

Adding the needed plugins

Make it even nicer (optional and opinionated)

Spring AI with Neo4j Vector Index

SpringAI

Neo4j Vector Store

Example

Setup

Application

Controller

Service

Run it

Creation of Documents with embeddings

Conversion

Conclusion

Cypher Language Server support for Neovim

First baby steps

Make it convenient to use

nvim-lspconfig

mason

mason-lspconfig

File type support

Conclusion

Maintain a simplistic blog

Copy blog posts (Single source of truth)

Scripting

Images

Folder structure

Jekyll configuration

Automation

Blog sources repository

The Jekyll repository

Conclusion

First experiences with DuckDB (and SQL in general after a while)

Disclaimer

Setup

Hello DuckDB

Activity table

Elevation Comparison and Type conversion

Querying

Table-less

Using Spring for GraphQL With Spring Data Neo4j

Introduction

Domain

Components

First Spring for GraphQL steps

Entity and Spring Data Neo4j setup

Spring for GraphQL setup

First run

Spring Data Neo4j GraphQL integration

Pagination

Let’s federate

Testing and test data

Neo4j-Migrations

Testcontainers

Testcontainers at development time

Topics left out / Try it

Summary

Using `yq` for metadata extraction