Fetching webmentions again. With Emacs this time!
I’ve reinstated webmention processing here and have semi-automated it with a pile of Emacs Lisp and a Makefile
, so I thought I’d write up some of the gory details.
Part 1 of… some?
You might have noticed, if you’re a regular visitor that webmentions have started showing up on the site again. I turned them off a while ago, but Aaron Parecki’s invaluable Webmention.io service has still been gathering them for me, so I’ve turned them back on. But in the mean time, I mislaid the code I was using to populate the necessary Hugo data files from Webmention. Exploratory code ahoy.
Start by faking it
I’m heavily indebted to Brian Wisti for his post, Using the Webmention.io API as the starting point to my explorations, but since I can’t be doing with Python, I used emacs.
I started with my very minor fork of restclient
# Grab the most recent 5 webmentions of bofh.org.uk
GET https://webmention.io/api/mentions.jf2?domain=bofh.org.uk&sort-dir=down&per-page=5&token=:token
Which produces the following JSON data:
Disclose this for the full wall of JSONDon’t say you weren’t warned!
{
"type": "feed",
"name": "Webmentions",
"children": [
{
"type": "entry",
"author": {
"type": "card",
"name": "Mike Spencer",
"photo": "https://avatars.webmention.io/fsn1.your-objectstorage.com/3ba5a0fdc660c44995fef428a601770fd0fe2619fc1f8c7e70ec7e9e1da66d4b.jpg",
"url": "https://mastodon.scot/@mikerspencer"
},
"url": "https://mendeddrum.org/@pdcawley/115162152891409560#favorited-by-109365771998686190",
"published": null,
"wm-received": "2025-09-07T09:35:22Z",
"wm-id": 1936897,
"wm-source": "https://brid.gy/like/mastodon/@pdcawley@mendeddrum.org/115162152891409560/109365771998686190",
"wm-target": "https://bofh.org.uk/note/8/",
"wm-protocol": "webmention",
"like-of": "https://bofh.org.uk/note/8/",
"wm-property": "like-of",
"wm-private": false
},
{
"type": "entry",
"author": {
"type": "card",
"name": "Jess Robinson",
"photo": "https://avatars.webmention.io/fsn1.your-objectstorage.com/a11ee0a58e54873140bf1f1965900378d866fce3fafae79e116b979bea3a8773.jpg",
"url": "https://fosstodon.org/@castaway"
},
"url": "https://mendeddrum.org/@pdcawley/115160370354067277#favorited-by-109562941096076318",
"published": null,
"wm-received": "2025-09-07T07:19:01Z",
"wm-id": 1936873,
"wm-source": "https://brid.gy/like/mastodon/@pdcawley@mendeddrum.org/115160370354067277/109562941096076318",
"wm-target": "https://bofh.org.uk/note/7/",
"wm-protocol": "webmention",
"like-of": "https://bofh.org.uk/note/7/",
"wm-property": "like-of",
"wm-private": false
},
{
"type": "entry",
"author": {
"type": "card",
"name": "Daniel Kelly Music",
"photo": "https://avatars.webmention.io/fsn1.your-objectstorage.com/7aa4815cbc1f993c0e2a6df03280dc168f5ad07fecd0313ddfc27eeb02e0b437.png",
"url": "https://aus.social/@yasslad"
},
"url": "https://mendeddrum.org/@pdcawley/115160370354067277#favorited-by-109307830089461078",
"published": null,
"wm-received": "2025-09-07T01:31:46Z",
"wm-id": 1936817,
"wm-source": "https://brid.gy/like/mastodon/@pdcawley@mendeddrum.org/115160370354067277/109307830089461078",
"wm-target": "https://bofh.org.uk/note/7/",
"wm-protocol": "webmention",
"like-of": "https://bofh.org.uk/note/7/",
"wm-property": "like-of",
"wm-private": false
},
{
"type": "entry",
"author": {
"type": "card",
"name": "Nick Anderson",
"photo": "https://avatars.webmention.io/fsn1.your-objectstorage.com/856b210b0770a292a4f9e76699c84aa3833f9252716ad3cab58b50a7b57205ee.jpg",
"url": "https://fosstodon.org/@nickanderson"
},
"url": "https://mendeddrum.org/@pdcawley/115107421781297473#favorited-by-109475479621313511",
"published": null,
"wm-received": "2025-08-28T21:48:52Z",
"wm-id": 1934308,
"wm-source": "https://brid.gy/like/mastodon/@pdcawley@mendeddrum.org/115107421781297473/109475479621313511",
"wm-target": "https://bofh.org.uk/note/6/",
"wm-protocol": "webmention",
"like-of": "https://bofh.org.uk/note/6/",
"wm-property": "like-of",
"wm-private": false
},
{
"type": "entry",
"author": {
"type": "card",
"name": "Nick Anderson",
"photo": "https://avatars.webmention.io/fsn1.your-objectstorage.com/856b210b0770a292a4f9e76699c84aa3833f9252716ad3cab58b50a7b57205ee.jpg",
"url": "https://fosstodon.org/@nickanderson"
},
"url": "https://fosstodon.org/@nickanderson/115108589417986943",
"published": "2025-08-28T21:48:05+00:00",
"wm-received": "2025-08-28T21:48:51Z",
"wm-id": 1934307,
"wm-source": "https://brid.gy/comment/mastodon/@pdcawley@mendeddrum.org/115107421781297473/115108589456284711",
"wm-target": "https://bofh.org.uk/note/6/",
"wm-protocol": "webmention",
"content": {
"html": "<p><span class=\"h-card\"><a href=\"https://mendeddrum.org/@pdcawley\" class=\"u-url\">@<span>pdcawley</span></a></span> I was gonna say, that looks like a source block within a source block which means there's another src block to make it render. Nice</p>",
"text": "@pdcawley I was gonna say, that looks like a source block within a source block which means there's another src block to make it render. Nice"
},
"in-reply-to": "https://bofh.org.uk/note/6/",
"wm-property": "in-reply-to",
"wm-private": false
}
]
}
The essential shape is something like this:
{ "type": "feed",
"name": "Webmentions",
"children": [
{ "wm-target": "https://bofh.org.uk/note/8/",
"wm-property": "like-of",
... },
{ "wm-target": "https://bofh.org.uk/note/8/",
"wm-property": "in-reply-to",
... },
... }]}
Restclient is great for interactively exploring a RESTful API, but it’s not so great for slicing and dicing the data in a Emacs-y way. I could sit down and learn jq
again, but I know Lisp, dammit, so after a frustrating hour or so trying to wrap my head around the default url-retrieve
interfaces, I went and grabbed the emacs-request
package instead because I found its API more comprehensible.
Because request
is a function where restclient
is more like an application running in Emacs, it’s way more useful for automating things. Here’s more or less the same request as above done in Lisp.
(let (response)
(request
"https://webmention.io/api/mentions.jf2"
:params `(("domain" . "bofh.org.uk")
("token" . ,wm-api-token)
("per-page" . "5")
("sort-dir" . "down"))
:parser 'json-parse-buffer
:sync t
:success (cl-function
(lambda (&key data &allow-other-keys)
(setq response data))))
response)
The code is obviously fiddlier, but it’s also programmable and, because we set an arbitrary :parser
function, it’s trivial to convert the returned JSON into a native Emacs lisp hash table
which looks a bit like this:
A wall of Lisp
#s(hash-table test equal data
("type" "feed" "name" "Webmentions" "children"
[#s(hash-table test equal data
("type" "entry" "author"
#s(hash-table test equal data
("type" "card" "name"
"Mike Spencer" "photo"
"https://avatars.webmention.io/fsn1.your-objectstorage.com/3ba5a0fdc660c44995fef428a601770fd0fe2619fc1f8c7e70ec7e9e1da66d4b.jpg"
"url"
"https://mastodon.scot/@mikerspencer"))
"url"
"https://mendeddrum.org/@pdcawley/115162152891409560#favorited-by-109365771998686190"
"published" :null "wm-received"
"2025-09-07T09:35:22Z" "wm-id" 1936897
"wm-source"
"https://brid.gy/like/mastodon/@pdcawley@mendeddrum.org/115162152891409560/109365771998686190"
"wm-target" "https://bofh.org.uk/note/8/"
"wm-protocol" "webmention" "like-of"
"https://bofh.org.uk/note/8/" "wm-property"
"like-of" "wm-private" :false))
#s(hash-table test equal data
("type" "entry" "author"
#s(hash-table test equal data
("type" "card" "name"
"Jess Robinson" "photo"
"https://avatars.webmention.io/fsn1.your-objectstorage.com/a11ee0a58e54873140bf1f1965900378d866fce3fafae79e116b979bea3a8773.jpg"
"url"
"https://fosstodon.org/@castaway"))
"url"
"https://mendeddrum.org/@pdcawley/115160370354067277#favorited-by-109562941096076318"
"published" :null "wm-received"
"2025-09-07T07:19:01Z" "wm-id" 1936873
"wm-source"
"https://brid.gy/like/mastodon/@pdcawley@mendeddrum.org/115160370354067277/109562941096076318"
"wm-target" "https://bofh.org.uk/note/7/"
"wm-protocol" "webmention" "like-of"
"https://bofh.org.uk/note/7/" "wm-property"
"like-of" "wm-private" :false))
#s(hash-table test equal data
("type" "entry" "author"
#s(hash-table test equal data
("type" "card" "name"
"Daniel Kelly Music" "photo"
"https://avatars.webmention.io/fsn1.your-objectstorage.com/7aa4815cbc1f993c0e2a6df03280dc168f5ad07fecd0313ddfc27eeb02e0b437.png"
"url"
"https://aus.social/@yasslad"))
"url"
"https://mendeddrum.org/@pdcawley/115160370354067277#favorited-by-109307830089461078"
"published" :null "wm-received"
"2025-09-07T01:31:46Z" "wm-id" 1936817
"wm-source"
"https://brid.gy/like/mastodon/@pdcawley@mendeddrum.org/115160370354067277/109307830089461078"
"wm-target" "https://bofh.org.uk/note/7/"
"wm-protocol" "webmention" "like-of"
"https://bofh.org.uk/note/7/" "wm-property"
"like-of" "wm-private" :false))
#s(hash-table test equal data
("type" "entry" "author"
#s(hash-table test equal data
("type" "card" "name"
"Nick Anderson" "photo"
"https://avatars.webmention.io/fsn1.your-objectstorage.com/856b210b0770a292a4f9e76699c84aa3833f9252716ad3cab58b50a7b57205ee.jpg"
"url"
"https://fosstodon.org/@nickanderson"))
"url"
"https://mendeddrum.org/@pdcawley/115107421781297473#favorited-by-109475479621313511"
"published" :null "wm-received"
"2025-08-28T21:48:52Z" "wm-id" 1934308
"wm-source"
"https://brid.gy/like/mastodon/@pdcawley@mendeddrum.org/115107421781297473/109475479621313511"
"wm-target" "https://bofh.org.uk/note/6/"
"wm-protocol" "webmention" "like-of"
"https://bofh.org.uk/note/6/" "wm-property"
"like-of" "wm-private" :false))
#s(hash-table test equal data
("type" "entry" "author"
#s(hash-table test equal data
("type" "card" "name"
"Nick Anderson" "photo"
"https://avatars.webmention.io/fsn1.your-objectstorage.com/856b210b0770a292a4f9e76699c84aa3833f9252716ad3cab58b50a7b57205ee.jpg"
"url"
"https://fosstodon.org/@nickanderson"))
"url"
"https://fosstodon.org/@nickanderson/115108589417986943"
"published" "2025-08-28T21:48:05+00:00"
"wm-received" "2025-08-28T21:48:51Z" "wm-id"
1934307 "wm-source"
"https://brid.gy/comment/mastodon/@pdcawley@mendeddrum.org/115107421781297473/115108589456284711"
"wm-target" "https://bofh.org.uk/note/6/"
"wm-protocol" "webmention" "content"
#s(hash-table test equal data
("html"
"<p><span class=\"h-card\"><a href=\"https://mendeddrum.org/@pdcawley\" class=\"u-url\">@<span>pdcawley</span></a></span> I was gonna say, that looks like a source block within a source block which means there's another src block to make it render. Nice</p>"
"text"
"@pdcawley I was gonna say, that looks like a source block within a source block which means there's another src block to make it render. Nice"))
"in-reply-to" "https://bofh.org.uk/note/6/"
"wm-property" "in-reply-to" "wm-private"
:false))]))
Verbose as hell, but something we can work with. Here’s a simplified alist representation which might be a little easier to understand.
'((type . "feed")
(name . "Webmentions")
(children
. [((wm-property . "like-of")
(wm-target . "https://bofh.org.uk/note/8/")
...)
((wm-property . "in-reply-to")
(wm-target . "https://bofh.org.uk/note/8/")
...)]))
The interesting stuff lives under the "children"
key, which we can get with (gethash "children" data)
.
To get all the webmentions for our domain, the Webmention API allows for pagination. We can ask for pages of, say 100 entries and if we get 100 entries back, append the result to our running collection of entries and request the next page. Once we get a result with fewer than 100 entries, we know we’re done and we can massage the data into a shape that Hugo can cope with
Then turn what we learn into a commands
Now we know what the data coming from Webmention.io looks like, and how we can page through it, let’s write a function, wm--fetch-all
to do that for us.
The (while more? ...)
loop keeps requesting more data until it gets a short response, at which point more?
becomes false and we return the accumulated all-entries
vector
And relax
We now have a handy list of all the webmentions relating to our site. The next step is to massage it into a data structure that will suit Hugo and export it as JSON files in the site’s data
directory. Which is a topic for another blog post, I think. If only because I’m reasonably sure that the data structure I’m currently using isn’t great.
I’ll hack it about a bit and report back.
You might have noticed, if you’re a regular visitor that webmentions have started showing up on the site again. I turned them off a while ago,
I turned off the home server that was handling the web hook calls from Webmention.io, planning to quickly move it and spin it up again. Ask me how that’s going.
but Aaron Parecki’s invaluable Webmention.io service has still been gathering them for me, so I’ve turned them back on. But in the mean time, I mislaid the code I was using to populate the necessary Hugo data files from Webmention. Exploratory code ahoy.Start by faking it
I’m heavily indebted to Brian Wisti for his post, Using the Webmention.io API as the starting point to my explorations, but since I can’t be doing with Python, I used emacs.
I started with my very minor fork of
restclient
Which produces the following JSON data:
Disclose this for the full wall of JSONDon’t say you weren’t warned!
The essential shape is something like this:
Restclient is great for interactively exploring a RESTful API, but it’s not so great for slicing and dicing the data in a Emacs-y way. I could sit down and learn
jq
again, but I know Lisp, dammit, so after a frustrating hour or so trying to wrap my head around the defaulturl-retrieve
interfaces, I went and grabbed theemacs-request
package instead because I found its API more comprehensible.Because
request
is a function whererestclient
is more like an application running in Emacs, it’s way more useful for automating things. Here’s more or less the same request as above done in Lisp.The code is obviously fiddlier, but it’s also programmable and, because we set an arbitrary
:parser
function, it’s trivial to convert the returned JSON into a native Emacs lisp hash tableIt’s not hard to generate an old school alist either, but that was annoyingly hard to serialise back to JSON, so I went with the default types because they seem to just work.
which looks a bit like this:A wall of LispVerbose as hell, but something we can work with. Here’s a simplified alist representation which might be a little easier to understand.
The interesting stuff lives under the
"children"
key, which we can get with(gethash "children" data)
.To get all the webmentions for our domain, the Webmention API allows for pagination. We can ask for pages of, say 100 entries and if we get 100 entries back, append the result to our running collection of entries and request the next page. Once we get a result with fewer than 100 entries, we know we’re done and we can massage the data into a shape that Hugo can cope with
Then turn what we learn into a commands
Now we know what the data coming from Webmention.io looks like, and how we can page through it, let’s write a function,
wm--fetch-all
to do that for us.The
(while more? ...)
loop keeps requesting more data until it gets a short response, at which pointmore?
becomes false and we return the accumulatedall-entries
vectorAnd relax
We now have a handy list of all the webmentions relating to our site. The next step is to massage it into a data structure that will suit Hugo and export it as JSON files in the site’s
data
directory. Which is a topic for another blog post, I think. If only because I’m reasonably sure that the data structure I’m currently using isn’t great.I’ll hack it about a bit and report back.