The beauty of using a static site generator to build your website is supposed to be that it’s all delightfully simple. Simple markdown formatted files go in at one end and a slim, fast and easy to serve website comes out the other end. All that remains is to upload those files to the appropriate directory on your server and all is well.

But never underestimate the ability of a long time Emacs user to complicate things. For instance, markdown is all well and good, but I’ve been doing most of my writing in Org Mode¹ so I really want to stay in Org mode to write these blog posts. Hugo understands .org files, so I could just lean on that, but the way Hugo treats org files seems slightly out of whack with what I think of as the Org way and I’d end up having to stick with the subset of org syntax that Hugo know. So I use ox-hugo, there’s a bit of configuration needed to make it work the way I like, but I prefer to change software to accommodate me rather than change me to accommodate software

I’ve had all that set up for a while. As I say, a tad fiddly at first, but once it’s in place, it just works.

Except… ox-hugo works by generating .md files from an org source, which are then used to generate the site, and I had things set up to autogenerate the html whenever I commited to the main branch of the blog repo, and the git server hook based system I was using only worked if those exported files were in repo.

That’s the sort of thing that makes me itch, because there were two files for any given article:

all-posts.org

the org file in which I write all my articles

article.md

the generated file that hugo uses to build the site.

The generated file is an artefact of the build process and simply repeats the info in the org file, which should be our single source of truth. It’s not a file that should be left around to be edited willy nilly because it could get out of sync with its source file. It’s certainly not the sort of file that should live in the repository.

I didn’t worry about this for ages, but it niggled at me. Then one day I read an article about using Github Actions to build an ox-hugo based site by installing emacs and ox-hugo on the VM that does the build step and generating the markdown files during the build by running Emacs

in batch mode. The markdown files never exist anywhere that anyone can edit them. So, of course I had to do that. Again, fiddly to set up, and arguably only of philosophical benefit, but worth it, I think.²

I could’ve left it there but the thing I miss about the old, slow, hard to maintain version of this site, is the sense of connection. The old site had comments, and pingback links to other blogs. There was a sense of connectedness that’s missing from a collection of articles. I want some of that back.

There is a way. In the time I’ve been mostly not blogging, some of the folks who kept at it have been cooking up a collection of tools, technologies and standards under the IndieWeb banner. There’s a whole suite of technologies involved, but the piece of the puzzle that I’m interested in right now is the WebMention, described as

… an @ mention that works across websites; so that you don’t feel immovable from Twitter or Fb

Roney Ngala (@rngala) on Twitter

Now we’re talking! It’s a really simple standard too. When you mention, like, comment on, repost, reply to, bookmark or simply publicly interact with an “h-entry”³ on the IndieWeb, you can send a webmention by sending a small chunk of JSON to the webmention endpoint of the entry you mentioned. Assuming all the content is marked up correctly, sending a webmention is delightfully easy. You can do it with curl, if that’s your thing, but I’m in an emacs buffer, so let’s use restclient

We mention https://indieweb.org in this post, so let’s find out its webmention endpoint.

HEAD https://indieweb.org

#+BEGIN_SRC html
<!-- HEAD https://indieweb.org -->
<!-- HTTP/1.1 200 OK -->
<!-- Server: nginx/1.24.0 -->
<!-- Date: Mon, 04 Mar 2024 13:42:00 GMT -->
<!-- Content-Type: text/html; charset=UTF-8 -->
<!-- Connection: keep-alive -->
<!-- Link: <https://webmention.io/indiewebcamp/webmention>; rel="webmention" -->
<!-- Cache-Control: no-cache -->
<!-- X-No-Cache: 1 -->
<!-- X-Cache: BYPASS -->
<!-- Request duration: 0.932852s -->
#+END_SRC

We’re looking for the 「Link: … ; rel="webmention"」 line. This tells us that to send a webmention targeting https://indieweb.org, we need to post it to https://webmention.io/indiewebcamp/webmention. Which is almost as simple as finding the end point. Here we go:

POST https://webmention.io/indiewebcamp/webmention
Content-Type: application/x-www-form-urlencoded

source=https://bofh.org.uk/2022/04/24/not-so-simple&target=https://indieweb.org

{
  "status": "queued",
  "summary": "Webmention was queued for processing",
  "location": "https://webmention.io/indiewebcamp/webmention/dELUcbvGREHxYZ2n1xqI",
  "source": "https://bofh.org.uk/2022/04/24/not-so-simple",
  "target": "https://indieweb.org"
}
// POST https://webmention.io/indiewebcamp/webmention
// HTTP/1.1 201 Created
// Content-Type: application/json;charset=UTF-8
// Content-Length: 236
// Connection: keep-alive
// Status: 201 Created
// Cache-Control: no-store
// Access-Control-Allow-Origin: *
// Location: https://webmention.io/indiewebcamp/webmention/dELUcbvGREHxYZ2n1xqI
// X-Content-Type-Options: nosniff
// Date: Mon, 04 Mar 2024 13:42:00 GMT
// X-Powered-By: Phusion Passenger 5.3.1
// Server: nginx/1.14.0 + Phusion Passenger 5.3.1
// Request duration: 0.489982s

The job is done, and we get a nice JSON formatted summary of what’s going on to boot.

Of course, if a webmention is so simple to send then it’s probably a pain in the bum to receive and it is… sort of. To receive a webmention request, you need to:

1. Run a web app to handle the request
2. Visit the source link
3. Parse out the microformats associated with the entry, its author and content
4. Figure out how to display the information

Steps 1–3 aren’t particularly hard, but they’re fiddly to get right and involve making web connections to potentially unsafe sites and I’m using Hugo to generate this site because I don’t want to be running potentially insecure code that’s exposed to the internet on a server that I own if I can possibly help it. Thankfully, I don’t have to. I can take a leaf out of indiweb.org’s book and just delegate that part to webmention.io. Webmention.io handles all that icky visiting of foreign websites and parsing out microformats for you and instead presents you with a feed consisting of all the webmention’s that’ve been sent to your site in a variety of formats. I’ve been consuming their .jf2 formatted feed for a while now. JF2 is a JSON representation of the microformats associated with the webmention’s source. Let’s grab something from that feed

GET https://webmention.io/api/mentions.jf2?per-page=2&page=0&sort-dir=up&target=https://bofh.org.uk/2013/03/10/in-which-piers-prepares-to-void-the-warranty/

{
  "type": "feed",
  "name": "Webmentions",
  "children": [
    {
      "type": "entry",
      "author": {
	"type": "card",
	"name": "David Gallows",
	"photo": "https://webmention.io/avatar/pbs.twimg.com/e7b750d847ffdcfc174845aadc9196125b83647258a1789bb2b92b493d223e8b.jpg",
	"url": "https://twitter.com/DavidGallows"
      },
      "url": "https://twitter.com/pdcawley/status/1517783526049001472#favorited-by-877428607",
      "published": null,
      "wm-received": "2022-04-23T09:59:17Z",
      "wm-id": 1385464,
      "wm-source": "https://brid.gy/like/twitter/pdcawley/1517783526049001472/877428607",
      "wm-target": "https://bofh.org.uk/2013/03/10/in-which-piers-prepares-to-void-the-warranty/",
      "wm-protocol": "webmention",
      "like-of": "https://bofh.org.uk/2013/03/10/in-which-piers-prepares-to-void-the-warranty/",
      "wm-property": "like-of",
      "wm-private": false
    },
    {
      "type": "entry",
      "author": {
	"type": "card",
	"name": "David Gallows",
	"photo": "https://webmention.io/avatar/pbs.twimg.com/e7b750d847ffdcfc174845aadc9196125b83647258a1789bb2b92b493d223e8b.jpg",
	"url": "https://twitter.com/DavidGallows"
      },
      "url": "https://twitter.com/DavidGallows/status/1517852498555555840",
      "published": "2022-04-23T13:06:50+00:00",
      "wm-received": "2022-04-23T15:12:04Z",
      "wm-id": 1385681,
      "wm-source": "https://brid.gy/comment/twitter/pdcawley/1517783526049001472/1517852498555555840",
      "wm-target": "https://bofh.org.uk/2013/03/10/in-which-piers-prepares-to-void-the-warranty/",
      "wm-protocol": "webmention",
      "content": {
	"html": "enjoyed reading about it :)\n\nI've been a trackball man since the word go, But have never been able to move away from Qwerty keyboards\n<a class=\"u-mention\" href=\"http://bofh.org.uk/\"></a>\n<a class=\"u-mention\" href=\"https://twitter.com/DrugCrazed\"></a>\n<a class=\"u-mention\" href=\"https://twitter.com/keyboardio\"></a>\n<a class=\"u-mention\" href=\"https://twitter.com/pdcawley\"></a>",
	"text": "enjoyed reading about it :)\n\nI've been a trackball man since the word go, But have never been able to move away from Qwerty keyboards"
      },
      "in-reply-to": "https://bofh.org.uk/2013/03/10/in-which-piers-prepares-to-void-the-warranty/",
      "wm-property": "in-reply-to",
      "wm-private": false
    }
  ]
}
// GET https://webmention.io/api/mentions.jf2?per-page=2&page=0&sort-dir=up&target=https://bofh.org.uk/2013/03/10/in-which-piers-prepares-to-void-the-warranty/
// HTTP/1.1 200 OK
// Content-Type: application/json;charset=UTF-8
// Content-Length: 2073
// Connection: keep-alive
// Status: 200 OK
// Cache-Control: no-store
// Access-Control-Allow-Origin: *
// X-Content-Type-Options: nosniff
// Date: Mon, 04 Mar 2024 13:42:00 GMT
// X-Powered-By: Phusion Passenger 5.3.1
// Server: nginx/1.14.0 + Phusion Passenger 5.3.1
// Request duration: 0.077520s

Lot’s of lovely structured data. Webmention.io has worked out that one mention was a like-of the blog post, and the other was in-reply-to it. We get details of the author of the mentioning post and, where appropriate, its content. If I wanted to run more Javascript on here (and I want to run less), I could attach a script which would consume the post’s feed and build a display of all of the mentions. It has a certain appeal, just add one script to the site and a dummy <div> or <ul> somewhere and I’m laughing. Plenty of sites do just that.

This is not one of those sites. It’s not even the first statically generated site to go down the route of statically generating a post’s webmentions. I was mostly inspired by Brian Wisti’s post about consuming the webmention.io API (except, of course, I don’t use any of his actual code.)

The site’s Github repo is configured so any commit on the main branch fires off a workflow that builds the site and ships all the files over to the webserver using rsync. If I take Brian’s idea for grabbing all my webmentions and ignore his warning about splitting it out into Hugo data files and just do it, I can start building the webmentions for posts. Huzzah!

It started so innocently I have a server here that hosts a couple of Docker images and one of them is N8N, a super powered, self-hosted open source replacement for IFTTT with all sorts of hooks into other services and a much more powerful Github client than the IFTTT offering. It’s a bit… JavaScript-y for my tastes, but you can’t have everything.

With a bit of fiddling, I had something that grabbed the webmention.io feed for the site every few hours, split it out into multiple files in data/mentions and updated GitHub. That’s what I was celebrating in We have WebMentions. I’ve moved on since then, because of course sorting out de-duplication and remembering information between runs of the script is annoyingly fiddly and full of edge cases. Basically, I ended up trying to emulate a proper database. Which is why the latest iteration of webmention handling uses a proper database. I would have used SQLite, but N8N doesn’t have a SQLite node available out of the box. It does have a PostgreSQL one though, and recent versions of that have really good JSON support. I’d tell you more, but wc-mode tells me I’m nearly 2500 words in to this article, I think I’ll wrap up for now and promise to give you the gory details in an upcoming article.