Fetching webmentions again. With Emacs this time!
I’ve reinstated webmention processing here and have semi-automated it with a pile of Emacs Lisp and a Makefile, so I thought I’d write up some of the gory details.
Part 1 of… some?
You might have noticed, if you’re a regular visitor that webmentions have started showing up on the site again. I turned them off a while ago, but Aaron Parecki’s invaluable Webmention.io service has still been gathering them for me, so I’ve turned them back on. But in the mean time, I mislaid the code I was using to populate the necessary Hugo data files from Webmention. Exploratory code ahoy.
Start by faking it
I’m heavily indebted to Brian Wisti for his post, Using the Webmention.io API as the starting point to my explorations, but since I can’t be doing with Python, I used emacs.
I started with my very minor fork of restclient
# Grab the most recent 5 webmentions of bofh.org.uk
GET https://webmention.io/api/mentions.jf2?domain=bofh.org.uk&sort-dir=down&per-page=5&token=:token
Which produces the following JSON data:
Disclose this for the full wall of JSONDon’t say you weren’t warned!
{
"type": "feed",
"name": "Webmentions",
"children": [
{
"type": "entry",
"author": {
"type": "card",
"name": "Mike Spencer",
"photo": "https://avatars.webmention.io/fsn1.your-objectstorage.com/3ba5a0fdc660c44995fef428a601770fd0fe2619fc1f8c7e70ec7e9e1da66d4b.jpg",
"url": "https://mastodon.scot/@mikerspencer"
},
"url": "https://mendeddrum.org/@pdcawley/115162152891409560#favorited-by-109365771998686190",
"published": null,
"wm-received": "2025-09-07T09:35:22Z",
"wm-id": 1936897,
"wm-source": "https://brid.gy/like/mastodon/@pdcawley@mendeddrum.org/115162152891409560/109365771998686190",
"wm-target": "https://bofh.org.uk/note/8/",
"wm-protocol": "webmention",
"like-of": "https://bofh.org.uk/note/8/",
"wm-property": "like-of",
"wm-private": false
},
{
"type": "entry",
"author": {
"type": "card",
"name": "Jess Robinson",
"photo": "https://avatars.webmention.io/fsn1.your-objectstorage.com/a11ee0a58e54873140bf1f1965900378d866fce3fafae79e116b979bea3a8773.jpg",
"url": "https://fosstodon.org/@castaway"
},
"url": "https://mendeddrum.org/@pdcawley/115160370354067277#favorited-by-109562941096076318",
"published": null,
"wm-received": "2025-09-07T07:19:01Z",
"wm-id": 1936873,
"wm-source": "https://brid.gy/like/mastodon/@pdcawley@mendeddrum.org/115160370354067277/109562941096076318",
"wm-target": "https://bofh.org.uk/note/7/",
"wm-protocol": "webmention",
"like-of": "https://bofh.org.uk/note/7/",
"wm-property": "like-of",
"wm-private": false
},
{
"type": "entry",
"author": {
"type": "card",
"name": "Daniel Kelly Music",
"photo": "https://avatars.webmention.io/fsn1.your-objectstorage.com/7aa4815cbc1f993c0e2a6df03280dc168f5ad07fecd0313ddfc27eeb02e0b437.png",
"url": "https://aus.social/@yasslad"
},
"url": "https://mendeddrum.org/@pdcawley/115160370354067277#favorited-by-109307830089461078",
"published": null,
"wm-received": "2025-09-07T01:31:46Z",
"wm-id": 1936817,
"wm-source": "https://brid.gy/like/mastodon/@pdcawley@mendeddrum.org/115160370354067277/109307830089461078",
"wm-target": "https://bofh.org.uk/note/7/",
"wm-protocol": "webmention",
"like-of": "https://bofh.org.uk/note/7/",
"wm-property": "like-of",
"wm-private": false
},
{
"type": "entry",
"author": {
"type": "card",
"name": "Nick Anderson",
"photo": "https://avatars.webmention.io/fsn1.your-objectstorage.com/856b210b0770a292a4f9e76699c84aa3833f9252716ad3cab58b50a7b57205ee.jpg",
"url": "https://fosstodon.org/@nickanderson"
},
"url": "https://mendeddrum.org/@pdcawley/115107421781297473#favorited-by-109475479621313511",
"published": null,
"wm-received": "2025-08-28T21:48:52Z",
"wm-id": 1934308,
"wm-source": "https://brid.gy/like/mastodon/@pdcawley@mendeddrum.org/115107421781297473/109475479621313511",
"wm-target": "https://bofh.org.uk/note/6/",
"wm-protocol": "webmention",
"like-of": "https://bofh.org.uk/note/6/",
"wm-property": "like-of",
"wm-private": false
},
{
"type": "entry",
"author": {
"type": "card",
"name": "Nick Anderson",
"photo": "https://avatars.webmention.io/fsn1.your-objectstorage.com/856b210b0770a292a4f9e76699c84aa3833f9252716ad3cab58b50a7b57205ee.jpg",
"url": "https://fosstodon.org/@nickanderson"
},
"url": "https://fosstodon.org/@nickanderson/115108589417986943",
"published": "2025-08-28T21:48:05+00:00",
"wm-received": "2025-08-28T21:48:51Z",
"wm-id": 1934307,
"wm-source": "https://brid.gy/comment/mastodon/@pdcawley@mendeddrum.org/115107421781297473/115108589456284711",
"wm-target": "https://bofh.org.uk/note/6/",
"wm-protocol": "webmention",
"content": {
"html": "<p><span class=\"h-card\"><a href=\"https://mendeddrum.org/@pdcawley\" class=\"u-url\">@<span>pdcawley</span></a></span> I was gonna say, that looks like a source block within a source block which means there's another src block to make it render. Nice</p>",
"text": "@pdcawley I was gonna say, that looks like a source block within a source block which means there's another src block to make it render. Nice"
},
"in-reply-to": "https://bofh.org.uk/note/6/",
"wm-property": "in-reply-to",
"wm-private": false
}
]
}
The essential shape is something like this:
{ "type": "feed",
"name": "Webmentions",
"children": [
{ "wm-target": "https://bofh.org.uk/note/8/",
"wm-property": "like-of",
... },
{ "wm-target": "https://bofh.org.uk/note/8/",
"wm-property": "in-reply-to",
... },
... }]}
Restclient is great for interactively exploring a RESTful API, but it’s not so great for slicing and dicing the data in a Emacs-y way. I could sit down and learn jq again, but I know Lisp, dammit, so after a frustrating hour or so trying to wrap my head around the default url-retrieve interfaces, I went and grabbed the emacs-request package instead because I found its API more comprehensible.
Because request is a function where restclient is more like an application running in Emacs, it’s way more useful for automating things. Here’s more or less the same request as above done in Lisp.
(let (response)
(request
"https://webmention.io/api/mentions.jf2"
:params `(("domain" . "bofh.org.uk")
("token" . ,wm-api-token)
("per-page" . "5")
("sort-dir" . "down"))
:parser 'json-parse-buffer
:sync t
:success (cl-function
(lambda (&key data &allow-other-keys)
(setq response data))))
response)
The code is obviously fiddlier, but it’s also programmable and, because we set an arbitrary :parser function, it’s trivial to convert the returned JSON into a native Emacs lisp hash table
which looks a bit like this:
A wall of Lisp
#s(hash-table test equal data
("type" "feed" "name" "Webmentions" "children"
[#s(hash-table test equal data
("type" "entry" "author"
#s(hash-table test equal data
("type" "card" "name"
"Mike Spencer" "photo"
"https://avatars.webmention.io/fsn1.your-objectstorage.com/3ba5a0fdc660c44995fef428a601770fd0fe2619fc1f8c7e70ec7e9e1da66d4b.jpg"
"url"
"https://mastodon.scot/@mikerspencer"))
"url"
"https://mendeddrum.org/@pdcawley/115162152891409560#favorited-by-109365771998686190"
"published" :null "wm-received"
"2025-09-07T09:35:22Z" "wm-id" 1936897
"wm-source"
"https://brid.gy/like/mastodon/@pdcawley@mendeddrum.org/115162152891409560/109365771998686190"
"wm-target" "https://bofh.org.uk/note/8/"
"wm-protocol" "webmention" "like-of"
"https://bofh.org.uk/note/8/" "wm-property"
"like-of" "wm-private" :false))
#s(hash-table test equal data
("type" "entry" "author"
#s(hash-table test equal data
("type" "card" "name"
"Jess Robinson" "photo"
"https://avatars.webmention.io/fsn1.your-objectstorage.com/a11ee0a58e54873140bf1f1965900378d866fce3fafae79e116b979bea3a8773.jpg"
"url"
"https://fosstodon.org/@castaway"))
"url"
"https://mendeddrum.org/@pdcawley/115160370354067277#favorited-by-109562941096076318"
"published" :null "wm-received"
"2025-09-07T07:19:01Z" "wm-id" 1936873
"wm-source"
"https://brid.gy/like/mastodon/@pdcawley@mendeddrum.org/115160370354067277/109562941096076318"
"wm-target" "https://bofh.org.uk/note/7/"
"wm-protocol" "webmention" "like-of"
"https://bofh.org.uk/note/7/" "wm-property"
"like-of" "wm-private" :false))
#s(hash-table test equal data
("type" "entry" "author"
#s(hash-table test equal data
("type" "card" "name"
"Daniel Kelly Music" "photo"
"https://avatars.webmention.io/fsn1.your-objectstorage.com/7aa4815cbc1f993c0e2a6df03280dc168f5ad07fecd0313ddfc27eeb02e0b437.png"
"url"
"https://aus.social/@yasslad"))
"url"
"https://mendeddrum.org/@pdcawley/115160370354067277#favorited-by-109307830089461078"
"published" :null "wm-received"
"2025-09-07T01:31:46Z" "wm-id" 1936817
"wm-source"
"https://brid.gy/like/mastodon/@pdcawley@mendeddrum.org/115160370354067277/109307830089461078"
"wm-target" "https://bofh.org.uk/note/7/"
"wm-protocol" "webmention" "like-of"
"https://bofh.org.uk/note/7/" "wm-property"
"like-of" "wm-private" :false))
#s(hash-table test equal data
("type" "entry" "author"
#s(hash-table test equal data
("type" "card" "name"
"Nick Anderson" "photo"
"https://avatars.webmention.io/fsn1.your-objectstorage.com/856b210b0770a292a4f9e76699c84aa3833f9252716ad3cab58b50a7b57205ee.jpg"
"url"
"https://fosstodon.org/@nickanderson"))
"url"
"https://mendeddrum.org/@pdcawley/115107421781297473#favorited-by-109475479621313511"
"published" :null "wm-received"
"2025-08-28T21:48:52Z" "wm-id" 1934308
"wm-source"
"https://brid.gy/like/mastodon/@pdcawley@mendeddrum.org/115107421781297473/109475479621313511"
"wm-target" "https://bofh.org.uk/note/6/"
"wm-protocol" "webmention" "like-of"
"https://bofh.org.uk/note/6/" "wm-property"
"like-of" "wm-private" :false))
#s(hash-table test equal data
("type" "entry" "author"
#s(hash-table test equal data
("type" "card" "name"
"Nick Anderson" "photo"
"https://avatars.webmention.io/fsn1.your-objectstorage.com/856b210b0770a292a4f9e76699c84aa3833f9252716ad3cab58b50a7b57205ee.jpg"
"url"
"https://fosstodon.org/@nickanderson"))
"url"
"https://fosstodon.org/@nickanderson/115108589417986943"
"published" "2025-08-28T21:48:05+00:00"
"wm-received" "2025-08-28T21:48:51Z" "wm-id"
1934307 "wm-source"
"https://brid.gy/comment/mastodon/@pdcawley@mendeddrum.org/115107421781297473/115108589456284711"
"wm-target" "https://bofh.org.uk/note/6/"
"wm-protocol" "webmention" "content"
#s(hash-table test equal data
("html"
"<p><span class=\"h-card\"><a href=\"https://mendeddrum.org/@pdcawley\" class=\"u-url\">@<span>pdcawley</span></a></span> I was gonna say, that looks like a source block within a source block which means there's another src block to make it render. Nice</p>"
"text"
"@pdcawley I was gonna say, that looks like a source block within a source block which means there's another src block to make it render. Nice"))
"in-reply-to" "https://bofh.org.uk/note/6/"
"wm-property" "in-reply-to" "wm-private"
:false))]))
Verbose as hell, but something we can work with. Here’s a simplified alist representation which might be a little easier to understand.
'((type . "feed")
(name . "Webmentions")
(children
. [((wm-property . "like-of")
(wm-target . "https://bofh.org.uk/note/8/")
...)
((wm-property . "in-reply-to")
(wm-target . "https://bofh.org.uk/note/8/")
...)]))
The interesting stuff lives under the "children" key, which we can get with (gethash "children" data).
To get all the webmentions for our domain, the Webmention API allows for pagination. We can ask for pages of, say 100 entries and if we get 100 entries back, append the result to our running collection of entries and request the next page. Once we get a result with fewer than 100 entries, we know we’re done and we can massage the data into a shape that Hugo can cope with
Then turn what we learn into a commands
Now we know what the data coming from Webmention.io looks like, and how we can page through it, let’s write a function, wm--fetch-all to do that for us.
(defun wm--fetch-all ()
"Fetch all the webmentions relating to our domain.
We fetch them 100 a time and return a vector. The domain of interest is grabbed
from the `WM_API_DOMAIN' environment variable, and the necessary Webmention API
token comes from `WM_API_TOKEN', also in the environment. The idea being that
these values don't sneak into a git repo and can be easily supplied by any CI
tools, or something like `direnv'.i"
(let ((all-entries (vector))
(page-size 100)
(wm-domain (or (getenv "WM_API_DOMAIN")
(error "WM_API_DOMAIN not in the environment")))
(wm-token (or (getenv "WM_API_TOKEN")
(error "WM_API_TOKEN not in the environment"))))
(let (entries
(page-index 0)
(more? t))
(while more?
(request
wm-webmention-endpoint
:params `(("domain" . ,wm-domain)
("token" . ,wm-token)
("page" . ,page-index)
("per-page" . ,page-size)
("sort-dir" . "up"))
:parser 'json-parse-buffer
:sync t
:success (cl-function
(lambda (&key data &allow-other-keys)
(if-let* ((entries (gethash "children" data (vector))))
(progn
(setq all-entries (vconcat all-entries entries)
more? (and entries (eql page-size (length entries)))
page-index (1+ page-index)))
(setq more? nil)))))
;; Be a good netizen
(sleep-for 1)))
all-entries))
The (while more? ...) loop keeps requesting more data until it gets a short response, at which point more? becomes false and we return the accumulated all-entries vector
And relax
We now have a handy list of all the webmentions relating to our site. The next step is to massage it into a data structure that will suit Hugo and export it as JSON files in the site’s data directory. Which is a topic for another blog post, I think. If only because I’m reasonably sure that the data structure I’m currently using isn’t great.
I’ll hack it about a bit and report back.
