Fetching webmentions again. With Emacs this time!
I’ve reinstated webmention processing here and have semi-automated it with a pile of Emacs Lisp and a Makefile, so I thought I’d write up some of the gory details.
Part 1 of… some?
You might have noticed, if you’re a regular visitor that webmentions have started showing up on the site again. I turned them off a while ago, but Aaron Parecki’s invaluable Webmention.io service has still been gathering them for me, so I’ve turned them back on. But in the mean time, I mislaid the code I was using to populate the necessary Hugo data files from Webmention. Exploratory code ahoy.
Start by faking it
I’m heavily indebted to Brian Wisti for his post, Using the Webmention.io API as the starting point to my explorations, but since I can’t be doing with Python, I used emacs.
I started with my very minor fork of restclient
# Grab the most recent 5 webmentions of bofh.org.uk
GET https://webmention.io/api/mentions.jf2?domain=bofh.org.uk&sort-dir=down&per-page=5&token=:token
Which produces the following JSON data:
Disclose this for the full wall of JSONDon’t say you weren’t warned!
{
"type": "feed",
"name": "Webmentions",
"children": [
{
"type": "entry",
"author": {
"type": "card",
"name": "Mike Spencer",
"photo": "https://avatars.webmention.io/fsn1.your-objectstorage.com/3ba5a0fdc660c44995fef428a601770fd0fe2619fc1f8c7e70ec7e9e1da66d4b.jpg",
"url": "https://mastodon.scot/@mikerspencer"
},
"url": "https://mendeddrum.org/@pdcawley/115162152891409560#favorited-by-109365771998686190",
"published": null,
"wm-received": "2025-09-07T09:35:22Z",
"wm-id": 1936897,
"wm-source": "https://brid.gy/like/mastodon/@pdcawley@mendeddrum.org/115162152891409560/109365771998686190",
"wm-target": "https://bofh.org.uk/note/8/",
"wm-protocol": "webmention",
"like-of": "https://bofh.org.uk/note/8/",
"wm-property": "like-of",
"wm-private": false
},
{
"type": "entry",
"author": {
"type": "card",
"name": "Jess Robinson",
"photo": "https://avatars.webmention.io/fsn1.your-objectstorage.com/a11ee0a58e54873140bf1f1965900378d866fce3fafae79e116b979bea3a8773.jpg",
"url": "https://fosstodon.org/@castaway"
},
"url": "https://mendeddrum.org/@pdcawley/115160370354067277#favorited-by-109562941096076318",
"published": null,
"wm-received": "2025-09-07T07:19:01Z",
"wm-id": 1936873,
"wm-source": "https://brid.gy/like/mastodon/@pdcawley@mendeddrum.org/115160370354067277/109562941096076318",
"wm-target": "https://bofh.org.uk/note/7/",
"wm-protocol": "webmention",
"like-of": "https://bofh.org.uk/note/7/",
"wm-property": "like-of",
"wm-private": false
},
{
"type": "entry",
"author": {
"type": "card",
"name": "Daniel Kelly Music",
"photo": "https://avatars.webmention.io/fsn1.your-objectstorage.com/7aa4815cbc1f993c0e2a6df03280dc168f5ad07fecd0313ddfc27eeb02e0b437.png",
"url": "https://aus.social/@yasslad"
},
"url": "https://mendeddrum.org/@pdcawley/115160370354067277#favorited-by-109307830089461078",
"published": null,
"wm-received": "2025-09-07T01:31:46Z",
"wm-id": 1936817,
"wm-source": "https://brid.gy/like/mastodon/@pdcawley@mendeddrum.org/115160370354067277/109307830089461078",
"wm-target": "https://bofh.org.uk/note/7/",
"wm-protocol": "webmention",
"like-of": "https://bofh.org.uk/note/7/",
"wm-property": "like-of",
"wm-private": false
},
{
"type": "entry",
"author": {
"type": "card",
"name": "Nick Anderson",
"photo": "https://avatars.webmention.io/fsn1.your-objectstorage.com/856b210b0770a292a4f9e76699c84aa3833f9252716ad3cab58b50a7b57205ee.jpg",
"url": "https://fosstodon.org/@nickanderson"
},
"url": "https://mendeddrum.org/@pdcawley/115107421781297473#favorited-by-109475479621313511",
"published": null,
"wm-received": "2025-08-28T21:48:52Z",
"wm-id": 1934308,
"wm-source": "https://brid.gy/like/mastodon/@pdcawley@mendeddrum.org/115107421781297473/109475479621313511",
"wm-target": "https://bofh.org.uk/note/6/",
"wm-protocol": "webmention",
"like-of": "https://bofh.org.uk/note/6/",
"wm-property": "like-of",
"wm-private": false
},
{
"type": "entry",
"author": {
"type": "card",
"name": "Nick Anderson",
"photo": "https://avatars.webmention.io/fsn1.your-objectstorage.com/856b210b0770a292a4f9e76699c84aa3833f9252716ad3cab58b50a7b57205ee.jpg",
"url": "https://fosstodon.org/@nickanderson"
},
"url": "https://fosstodon.org/@nickanderson/115108589417986943",
"published": "2025-08-28T21:48:05+00:00",
"wm-received": "2025-08-28T21:48:51Z",
"wm-id": 1934307,
"wm-source": "https://brid.gy/comment/mastodon/@pdcawley@mendeddrum.org/115107421781297473/115108589456284711",
"wm-target": "https://bofh.org.uk/note/6/",
"wm-protocol": "webmention",
"content": {
"html": "<p><span class=\"h-card\"><a href=\"https://mendeddrum.org/@pdcawley\" class=\"u-url\">@<span>pdcawley</span></a></span> I was gonna say, that looks like a source block within a source block which means there's another src block to make it render. Nice</p>",
"text": "@pdcawley I was gonna say, that looks like a source block within a source block which means there's another src block to make it render. Nice"
},
"in-reply-to": "https://bofh.org.uk/note/6/",
"wm-property": "in-reply-to",
"wm-private": false
}
]
}
The essential shape is something like this:
{ "type": "feed",
"name": "Webmentions",
"children": [
{ "wm-target": "https://bofh.org.uk/note/8/",
"wm-property": "like-of",
... },
{ "wm-target": "https://bofh.org.uk/note/8/",
"wm-property": "in-reply-to",
... },
... }]}
Restclient is great for interactively exploring a RESTful API, but it’s not so great for slicing and dicing the data in a Emacs-y way. I could sit down and learn jq again, but I know Lisp, dammit, so after a frustrating hour or so trying to wrap my head around the default url-retrieve interfaces, I went and grabbed the emacs-request package instead because I found its API more comprehensible.
Because request is a function where restclient is more like an application running in Emacs, it’s way more useful for automating things. Here’s more or less the same request as above done in Lisp.
(let (response)
(request
"https://webmention.io/api/mentions.jf2"
:params `(("domain" . "bofh.org.uk")
("token" . ,wm-api-token)
("per-page" . "5")
("sort-dir" . "down"))
:parser 'json-parse-buffer
:sync t
:success (cl-function
(lambda (&key data &allow-other-keys)
(setq response data))))
response)
The code is obviously fiddlier, but it’s also programmable and, because we set an arbitrary :parser function, it’s trivial to convert the returned JSON into a native Emacs lisp hash table
which looks a bit like this:
A wall of Lisp
#s(hash-table test equal data
("type" "feed" "name" "Webmentions" "children"
[#s(hash-table test equal data
("type" "entry" "author"
#s(hash-table test equal data
("type" "card" "name"
"Mike Spencer" "photo"
"https://avatars.webmention.io/fsn1.your-objectstorage.com/3ba5a0fdc660c44995fef428a601770fd0fe2619fc1f8c7e70ec7e9e1da66d4b.jpg"
"url"
"https://mastodon.scot/@mikerspencer"))
"url"
"https://mendeddrum.org/@pdcawley/115162152891409560#favorited-by-109365771998686190"
"published" :null "wm-received"
"2025-09-07T09:35:22Z" "wm-id" 1936897
"wm-source"
"https://brid.gy/like/mastodon/@pdcawley@mendeddrum.org/115162152891409560/109365771998686190"
"wm-target" "https://bofh.org.uk/note/8/"
"wm-protocol" "webmention" "like-of"
"https://bofh.org.uk/note/8/" "wm-property"
"like-of" "wm-private" :false))
#s(hash-table test equal data
("type" "entry" "author"
#s(hash-table test equal data
("type" "card" "name"
"Jess Robinson" "photo"
"https://avatars.webmention.io/fsn1.your-objectstorage.com/a11ee0a58e54873140bf1f1965900378d866fce3fafae79e116b979bea3a8773.jpg"
"url"
"https://fosstodon.org/@castaway"))
"url"
"https://mendeddrum.org/@pdcawley/115160370354067277#favorited-by-109562941096076318"
"published" :null "wm-received"
"2025-09-07T07:19:01Z" "wm-id" 1936873
"wm-source"
"https://brid.gy/like/mastodon/@pdcawley@mendeddrum.org/115160370354067277/109562941096076318"
"wm-target" "https://bofh.org.uk/note/7/"
"wm-protocol" "webmention" "like-of"
"https://bofh.org.uk/note/7/" "wm-property"
"like-of" "wm-private" :false))
#s(hash-table test equal data
("type" "entry" "author"
#s(hash-table test equal data
("type" "card" "name"
"Daniel Kelly Music" "photo"
"https://avatars.webmention.io/fsn1.your-objectstorage.com/7aa4815cbc1f993c0e2a6df03280dc168f5ad07fecd0313ddfc27eeb02e0b437.png"
"url"
"https://aus.social/@yasslad"))
"url"
"https://mendeddrum.org/@pdcawley/115160370354067277#favorited-by-109307830089461078"
"published" :null "wm-received"
"2025-09-07T01:31:46Z" "wm-id" 1936817
"wm-source"
"https://brid.gy/like/mastodon/@pdcawley@mendeddrum.org/115160370354067277/109307830089461078"
"wm-target" "https://bofh.org.uk/note/7/"
"wm-protocol" "webmention" "like-of"
"https://bofh.org.uk/note/7/" "wm-property"
"like-of" "wm-private" :false))
#s(hash-table test equal data
("type" "entry" "author"
#s(hash-table test equal data
("type" "card" "name"
"Nick Anderson" "photo"
"https://avatars.webmention.io/fsn1.your-objectstorage.com/856b210b0770a292a4f9e76699c84aa3833f9252716ad3cab58b50a7b57205ee.jpg"
"url"
"https://fosstodon.org/@nickanderson"))
"url"
"https://mendeddrum.org/@pdcawley/115107421781297473#favorited-by-109475479621313511"
"published" :null "wm-received"
"2025-08-28T21:48:52Z" "wm-id" 1934308
"wm-source"
"https://brid.gy/like/mastodon/@pdcawley@mendeddrum.org/115107421781297473/109475479621313511"
"wm-target" "https://bofh.org.uk/note/6/"
"wm-protocol" "webmention" "like-of"
"https://bofh.org.uk/note/6/" "wm-property"
"like-of" "wm-private" :false))
#s(hash-table test equal data
("type" "entry" "author"
#s(hash-table test equal data
("type" "card" "name"
"Nick Anderson" "photo"
"https://avatars.webmention.io/fsn1.your-objectstorage.com/856b210b0770a292a4f9e76699c84aa3833f9252716ad3cab58b50a7b57205ee.jpg"
"url"
"https://fosstodon.org/@nickanderson"))
"url"
"https://fosstodon.org/@nickanderson/115108589417986943"
"published" "2025-08-28T21:48:05+00:00"
"wm-received" "2025-08-28T21:48:51Z" "wm-id"
1934307 "wm-source"
"https://brid.gy/comment/mastodon/@pdcawley@mendeddrum.org/115107421781297473/115108589456284711"
"wm-target" "https://bofh.org.uk/note/6/"
"wm-protocol" "webmention" "content"
#s(hash-table test equal data
("html"
"<p><span class=\"h-card\"><a href=\"https://mendeddrum.org/@pdcawley\" class=\"u-url\">@<span>pdcawley</span></a></span> I was gonna say, that looks like a source block within a source block which means there's another src block to make it render. Nice</p>"
"text"
"@pdcawley I was gonna say, that looks like a source block within a source block which means there's another src block to make it render. Nice"))
"in-reply-to" "https://bofh.org.uk/note/6/"
"wm-property" "in-reply-to" "wm-private"
:false))]))
Verbose as hell, but something we can work with. Here’s a simplified alist representation which might be a little easier to understand.
'((type . "feed")
(name . "Webmentions")
(children
. [((wm-property . "like-of")
(wm-target . "https://bofh.org.uk/note/8/")
...)
((wm-property . "in-reply-to")
(wm-target . "https://bofh.org.uk/note/8/")
...)]))
The interesting stuff lives under the "children" key, which we can get with (gethash "children" data).
To get all the webmentions for our domain, the Webmention API allows for pagination. We can ask for pages of, say 100 entries and if we get 100 entries back, append the result to our running collection of entries and request the next page. Once we get a result with fewer than 100 entries, we know we’re done and we can massage the data into a shape that Hugo can cope with
Then turn what we learn into a commands
Now we know what the data coming from Webmention.io looks like, and how we can page through it, let’s write a function, wm--fetch-all to do that for us.
The (while more? ...) loop keeps requesting more data until it gets a short response, at which point more? becomes false and we return the accumulated all-entries vector
And relax
We now have a handy list of all the webmentions relating to our site. The next step is to massage it into a data structure that will suit Hugo and export it as JSON files in the site’s data directory. Which is a topic for another blog post, I think. If only because I’m reasonably sure that the data structure I’m currently using isn’t great.
I’ll hack it about a bit and report back.

When last we left off we had worked out how to grab all the mentions of this site that Webmentions.io knew about and now we want to write that out to the
data/directory in a format that’s easy to deal with in Hugo.If you’ve read my earlier note, you’ll know that I’ve been evolving the data schema towards something that’s easy for Hugo to deal with and reasonably comprehensible for me too.
As things currently stand, I’ve settled on dropping all the mentions in a single file,
data/mentions/all.jsonI’d rather use
which is structured along these lines:data/mentions.json, but Hugo’s data system doesn’t seem to pick that up, so I’ll live with the slightly more clunky option.Just imagine that JSON object has a bunch more paths as keys referencing similar objects keyed by mention type.
As it stands,
wm--fetch-allis returning a flat sequence of webmention objects that we want to process into a more structured object,JSON/Javascript calls them objects, old Perl heads like me think of them as hashes, and they’re “Hash Tables” in Emacs Lisp. I’ll be calling them hashes from now on.
in other words we want to fold (or “reduce” in Emacs Lisp terminology) the sequence into a hash. And I know just the function for that. Let’s see whatdescribe-functionhas to say aboutseq-reduce:So we can write
and that will handle the business of iterating over the sequence of mentions for us, and all we have to do is write
wm--add-mention-to-hash-tableto populate the hash we made with(make-hash-table :test 'equal)We need to use that
one mention at a time, and return the modified hash (You and I both know that it’s the same old hash mutated, but let’s pretend it isn’t, eh?).:test ’equalpart becausejson-insertwants a hash with strings as keys and the default hash returned by(make-hash-table)compares keys usingeqlwhich might or might not work when comparing strings. Not a problem whichequalhas.What does that function look like? Here’s what I wrote:
We grab the path from the
"wm-target"key, which is actually a URL rather than a simple pathWe could just use the URL, and that would work fine on this site, but not when I’m running on localhost. The path will always match with
so, rather than writing.RelPermalink, but the host part of.Permalinkis different in development than in production.we’ll thread
mentionthrough that series of transformations usingdash.el’s threading macro,-->.We use the path to grab
mentions-hashfrom theacc-umulating hash and, if there isn’t already one there, we grab an empty, but structured hash usingwm-new-mentions-hash, which looks like this:Now we look up
"wm-property"inmention, and use that to grab its associated vector of mentions. Well, we would, but there’s a small wrinkle.We’re only currently interested in four kinds of mention, but Webmention.io doesn’t know that. We could throw the extras away, but what if we became interested in
bookmark-ofmentions or whatever somewhere down the road. So let’s collect them under theotherkey. Which is where this hacky section of ourlet*form comes in:What’s going on here then?
First, we make a guess at the
mention-typewe’re going to file the current mention under by grabbing the"wm-property"and use that value to lookup the mention type inmentions-hash. If it’s one of the four types we’re interested in, that will be a vector, which is truth-y, otherwise we getnil, which is false-y so we change the mention type to “other” and grab that vector from the mention hash.We now know the key path we’re going to store our mentions in, and we have the current vector of mentions associated with it. So, if we already know
(seq-contains mentions mention)about the current mention, we reuse that, otherwise we make a new vector with the current mention added to it.That done, it’s a simple matter of putting the new mentions vector into our mentions hash, putting the mentions hash into our accumulating hash and returning that.
With that done, it’s a simple matter of opening
data/mentions/all.json, erasing the buffer, calling(json-insert (seq-reduce ...))to update the data and saving it. Here’s the code which does exactly that.Over in the Hugo partial that renders the bit of the page immediately after this, we can get at the data like this:
I’ll leave the rest as an exercise for the interested reader. However, I will note that the Webmention.io API includes the option to pass in a
sinceargument, so it wouldn’t be hard to writewithout having to change our reducing function at all.
Separation of concerns, baby! Separation of concerns!