Fetching webmentions again. With Emacs this time!

Piers Cawley 2025-09-072025-09-07

I’ve reinstated webmention processing here and have semi-automated it with a pile of Emacs Lisp and a Makefile, so I thought I’d write up some of the gory details.

Part 1 of… some?

You might have noticed, if you’re a regular visitor that webmentions have started showing up on the site again. I turned them off a while ago, but Aaron Parecki’s invaluable Webmention.io service has still been gathering them for me, so I’ve turned them back on. But in the mean time, I mislaid the code I was using to populate the necessary Hugo data files from Webmention. Exploratory code ahoy.

Start by faking it

I’m heavily indebted to Brian Wisti for his post, Using the Webmention.io API as the starting point to my explorations, but since I can’t be doing with Python, I used emacs.

I started with my very minor fork of restclient

# Grab the most recent 5 webmentions of bofh.org.uk
GET https://webmention.io/api/mentions.jf2?domain=bofh.org.uk&sort-dir=down&per-page=5&token=:token

Which produces the following JSON data:

Disclose this for the full wall of JSON

Don’t say you weren’t warned!

{
  "type": "feed",
  "name": "Webmentions",
  "children": [
    {
      "type": "entry",
      "author": {
        "type": "card",
        "name": "Mike Spencer",
        "photo": "https://avatars.webmention.io/fsn1.your-objectstorage.com/3ba5a0fdc660c44995fef428a601770fd0fe2619fc1f8c7e70ec7e9e1da66d4b.jpg",
        "url": "https://mastodon.scot/@mikerspencer"
      },
      "url": "https://mendeddrum.org/@pdcawley/115162152891409560#favorited-by-109365771998686190",
      "published": null,
      "wm-received": "2025-09-07T09:35:22Z",
      "wm-id": 1936897,
      "wm-source": "https://brid.gy/like/mastodon/@pdcawley@mendeddrum.org/115162152891409560/109365771998686190",
      "wm-target": "https://bofh.org.uk/note/8/",
      "wm-protocol": "webmention",
      "like-of": "https://bofh.org.uk/note/8/",
      "wm-property": "like-of",
      "wm-private": false
    },
    {
      "type": "entry",
      "author": {
        "type": "card",
        "name": "Jess Robinson",
        "photo": "https://avatars.webmention.io/fsn1.your-objectstorage.com/a11ee0a58e54873140bf1f1965900378d866fce3fafae79e116b979bea3a8773.jpg",
        "url": "https://fosstodon.org/@castaway"
      },
      "url": "https://mendeddrum.org/@pdcawley/115160370354067277#favorited-by-109562941096076318",
      "published": null,
      "wm-received": "2025-09-07T07:19:01Z",
      "wm-id": 1936873,
      "wm-source": "https://brid.gy/like/mastodon/@pdcawley@mendeddrum.org/115160370354067277/109562941096076318",
      "wm-target": "https://bofh.org.uk/note/7/",
      "wm-protocol": "webmention",
      "like-of": "https://bofh.org.uk/note/7/",
      "wm-property": "like-of",
      "wm-private": false
    },
    {
      "type": "entry",
      "author": {
        "type": "card",
        "name": "Daniel Kelly Music",
        "photo": "https://avatars.webmention.io/fsn1.your-objectstorage.com/7aa4815cbc1f993c0e2a6df03280dc168f5ad07fecd0313ddfc27eeb02e0b437.png",
        "url": "https://aus.social/@yasslad"
      },
      "url": "https://mendeddrum.org/@pdcawley/115160370354067277#favorited-by-109307830089461078",
      "published": null,
      "wm-received": "2025-09-07T01:31:46Z",
      "wm-id": 1936817,
      "wm-source": "https://brid.gy/like/mastodon/@pdcawley@mendeddrum.org/115160370354067277/109307830089461078",
      "wm-target": "https://bofh.org.uk/note/7/",
      "wm-protocol": "webmention",
      "like-of": "https://bofh.org.uk/note/7/",
      "wm-property": "like-of",
      "wm-private": false
    },
    {
      "type": "entry",
      "author": {
        "type": "card",
        "name": "Nick Anderson",
        "photo": "https://avatars.webmention.io/fsn1.your-objectstorage.com/856b210b0770a292a4f9e76699c84aa3833f9252716ad3cab58b50a7b57205ee.jpg",
        "url": "https://fosstodon.org/@nickanderson"
      },
      "url": "https://mendeddrum.org/@pdcawley/115107421781297473#favorited-by-109475479621313511",
      "published": null,
      "wm-received": "2025-08-28T21:48:52Z",
      "wm-id": 1934308,
      "wm-source": "https://brid.gy/like/mastodon/@pdcawley@mendeddrum.org/115107421781297473/109475479621313511",
      "wm-target": "https://bofh.org.uk/note/6/",
      "wm-protocol": "webmention",
      "like-of": "https://bofh.org.uk/note/6/",
      "wm-property": "like-of",
      "wm-private": false
    },
    {
      "type": "entry",
      "author": {
        "type": "card",
        "name": "Nick Anderson",
        "photo": "https://avatars.webmention.io/fsn1.your-objectstorage.com/856b210b0770a292a4f9e76699c84aa3833f9252716ad3cab58b50a7b57205ee.jpg",
        "url": "https://fosstodon.org/@nickanderson"
      },
      "url": "https://fosstodon.org/@nickanderson/115108589417986943",
      "published": "2025-08-28T21:48:05+00:00",
      "wm-received": "2025-08-28T21:48:51Z",
      "wm-id": 1934307,
      "wm-source": "https://brid.gy/comment/mastodon/@pdcawley@mendeddrum.org/115107421781297473/115108589456284711",
      "wm-target": "https://bofh.org.uk/note/6/",
      "wm-protocol": "webmention",
      "content": {
        "html": "<p><span class=\"h-card\"><a href=\"https://mendeddrum.org/@pdcawley\" class=\"u-url\">@<span>pdcawley</span></a></span> I was gonna say, that looks like a source block within a source block which means there's another src block to make it render. Nice</p>",
        "text": "@pdcawley I was gonna say, that looks like a source block within a source block which means there's another src block to make it render. Nice"
      },
      "in-reply-to": "https://bofh.org.uk/note/6/",
      "wm-property": "in-reply-to",
      "wm-private": false
    }
  ]
}

The essential shape is something like this:

{ "type": "feed",
  "name": "Webmentions",
  "children": [
    { "wm-target": "https://bofh.org.uk/note/8/",
      "wm-property": "like-of",
      ... },
    { "wm-target": "https://bofh.org.uk/note/8/",
      "wm-property": "in-reply-to",
      ... },
    ... }]}

Restclient is great for interactively exploring a RESTful API, but it’s not so great for slicing and dicing the data in a Emacs-y way. I could sit down and learn jq again, but I know Lisp, dammit, so after a frustrating hour or so trying to wrap my head around the default url-retrieve interfaces, I went and grabbed the emacs-request package instead because I found its API more comprehensible.

Because request is a function where restclient is more like an application running in Emacs, it’s way more useful for automating things. Here’s more or less the same request as above done in Lisp.

(let (response)
  (request
    "https://webmention.io/api/mentions.jf2"
    :params `(("domain" . "bofh.org.uk")
              ("token" . ,wm-api-token)
              ("per-page" . "5")
              ("sort-dir" . "down"))
    :parser 'json-parse-buffer
    :sync t
    :success (cl-function
              (lambda (&key data &allow-other-keys)
                (setq response data))))
  response)

The code is obviously fiddlier, but it’s also programmable and, because we set an arbitrary :parser function, it’s trivial to convert the returned JSON into a native Emacs lisp hash table which looks a bit like this:

A wall of Lisp

#s(hash-table test equal data
              ("type" "feed" "name" "Webmentions" "children"
               [#s(hash-table test equal data
                              ("type" "entry" "author"
                               #s(hash-table test equal data
                                             ("type" "card" "name"
                                              "Mike Spencer" "photo"
                                              "https://avatars.webmention.io/fsn1.your-objectstorage.com/3ba5a0fdc660c44995fef428a601770fd0fe2619fc1f8c7e70ec7e9e1da66d4b.jpg"
                                              "url"
                                              "https://mastodon.scot/@mikerspencer"))
                               "url"
                               "https://mendeddrum.org/@pdcawley/115162152891409560#favorited-by-109365771998686190"
                               "published" :null "wm-received"
                               "2025-09-07T09:35:22Z" "wm-id" 1936897
                               "wm-source"
                               "https://brid.gy/like/mastodon/@pdcawley@mendeddrum.org/115162152891409560/109365771998686190"
                               "wm-target" "https://bofh.org.uk/note/8/"
                               "wm-protocol" "webmention" "like-of"
                               "https://bofh.org.uk/note/8/" "wm-property"
                               "like-of" "wm-private" :false))
                  #s(hash-table test equal data
                                ("type" "entry" "author"
                                 #s(hash-table test equal data
                                               ("type" "card" "name"
                                                "Jess Robinson" "photo"
                                                "https://avatars.webmention.io/fsn1.your-objectstorage.com/a11ee0a58e54873140bf1f1965900378d866fce3fafae79e116b979bea3a8773.jpg"
                                                "url"
                                                "https://fosstodon.org/@castaway"))
                                 "url"
                                 "https://mendeddrum.org/@pdcawley/115160370354067277#favorited-by-109562941096076318"
                                 "published" :null "wm-received"
                                 "2025-09-07T07:19:01Z" "wm-id" 1936873
                                 "wm-source"
                                 "https://brid.gy/like/mastodon/@pdcawley@mendeddrum.org/115160370354067277/109562941096076318"
                                 "wm-target" "https://bofh.org.uk/note/7/"
                                 "wm-protocol" "webmention" "like-of"
                                 "https://bofh.org.uk/note/7/" "wm-property"
                                 "like-of" "wm-private" :false))
                  #s(hash-table test equal data
                                ("type" "entry" "author"
                                 #s(hash-table test equal data
                                               ("type" "card" "name"
                                                "Daniel Kelly Music" "photo"
                                                "https://avatars.webmention.io/fsn1.your-objectstorage.com/7aa4815cbc1f993c0e2a6df03280dc168f5ad07fecd0313ddfc27eeb02e0b437.png"
                                                "url"
                                                "https://aus.social/@yasslad"))
                                 "url"
                                 "https://mendeddrum.org/@pdcawley/115160370354067277#favorited-by-109307830089461078"
                                 "published" :null "wm-received"
                                 "2025-09-07T01:31:46Z" "wm-id" 1936817
                                 "wm-source"
                                 "https://brid.gy/like/mastodon/@pdcawley@mendeddrum.org/115160370354067277/109307830089461078"
                                 "wm-target" "https://bofh.org.uk/note/7/"
                                 "wm-protocol" "webmention" "like-of"
                                 "https://bofh.org.uk/note/7/" "wm-property"
                                 "like-of" "wm-private" :false))
                  #s(hash-table test equal data
                                ("type" "entry" "author"
                                 #s(hash-table test equal data
                                               ("type" "card" "name"
                                                "Nick Anderson" "photo"
                                                "https://avatars.webmention.io/fsn1.your-objectstorage.com/856b210b0770a292a4f9e76699c84aa3833f9252716ad3cab58b50a7b57205ee.jpg"
                                                "url"
                                                "https://fosstodon.org/@nickanderson"))
                                 "url"
                                 "https://mendeddrum.org/@pdcawley/115107421781297473#favorited-by-109475479621313511"
                                 "published" :null "wm-received"
                                 "2025-08-28T21:48:52Z" "wm-id" 1934308
                                 "wm-source"
                                 "https://brid.gy/like/mastodon/@pdcawley@mendeddrum.org/115107421781297473/109475479621313511"
                                 "wm-target" "https://bofh.org.uk/note/6/"
                                 "wm-protocol" "webmention" "like-of"
                                 "https://bofh.org.uk/note/6/" "wm-property"
                                 "like-of" "wm-private" :false))
                  #s(hash-table test equal data
                                ("type" "entry" "author"
                                 #s(hash-table test equal data
                                               ("type" "card" "name"
                                                "Nick Anderson" "photo"
                                                "https://avatars.webmention.io/fsn1.your-objectstorage.com/856b210b0770a292a4f9e76699c84aa3833f9252716ad3cab58b50a7b57205ee.jpg"
                                                "url"
                                                "https://fosstodon.org/@nickanderson"))
                                 "url"
                                 "https://fosstodon.org/@nickanderson/115108589417986943"
                                 "published" "2025-08-28T21:48:05+00:00"
                                 "wm-received" "2025-08-28T21:48:51Z" "wm-id"
                                 1934307 "wm-source"
                                 "https://brid.gy/comment/mastodon/@pdcawley@mendeddrum.org/115107421781297473/115108589456284711"
                                 "wm-target" "https://bofh.org.uk/note/6/"
                                 "wm-protocol" "webmention" "content"
                                 #s(hash-table test equal data
                                               ("html"
                                                "<p><span class=\"h-card\"><a href=\"https://mendeddrum.org/@pdcawley\" class=\"u-url\">@<span>pdcawley</span></a></span> I was gonna say, that looks like a source block within a source block which means there's another src block to make it render. Nice</p>"
                                                "text"
                                                "@pdcawley I was gonna say, that looks like a source block within a source block which means there's another src block to make it render. Nice"))
                                 "in-reply-to" "https://bofh.org.uk/note/6/"
                                 "wm-property" "in-reply-to" "wm-private"
                                 :false))]))

Verbose as hell, but something we can work with. Here’s a simplified alist representation which might be a little easier to understand.

'((type . "feed")
  (name . "Webmentions")
  (children
   . [((wm-property . "like-of")
       (wm-target . "https://bofh.org.uk/note/8/")
       ...)
      ((wm-property . "in-reply-to")
       (wm-target . "https://bofh.org.uk/note/8/")
       ...)]))

The interesting stuff lives under the "children" key, which we can get with (gethash "children" data).

To get all the webmentions for our domain, the Webmention API allows for pagination. We can ask for pages of, say 100 entries and if we get 100 entries back, append the result to our running collection of entries and request the next page. Once we get a result with fewer than 100 entries, we know we’re done and we can massage the data into a shape that Hugo can cope with

Then turn what we learn into a commands

Now we know what the data coming from Webmention.io looks like, and how we can page through it, let’s write a function, wm--fetch-all to do that for us.

The (while more? ...) loop keeps requesting more data until it gets a short response, at which point more? becomes false and we return the accumulated all-entries vector

And relax

We now have a handy list of all the webmentions relating to our site. The next step is to massage it into a data structure that will suit Hugo and export it as JSON files in the site’s data directory. Which is a topic for another blog post, I think. If only because I’m reasonably sure that the data structure I’m currently using isn’t great.

I’ll hack it about a bit and report back.

Likes

Reposts

Other Mentions

Piers Cawley

When last we left off we had worked out how to grab all the mentions of this site that Webmentions.io knew about and now we want to write that out to the data/ directory in a format that’s easy to deal with in Hugo.

If you’ve read my earlier note, you’ll know that I’ve been evolving the data schema towards something that’s easy for Hugo to deal with and reasonably comprehensible for me too.

As things currently stand, I’ve settled on dropping all the mentions in a single file, data/mentions/all.json

I’d rather use data/mentions.json, but Hugo’s data system doesn’t seem to pick that up, so I’ll live with the slightly more clunky option.

which is structured along these lines:

{
  "/note/7/": {
    "like-of": [
      {
        "type": "entry",
        "author": {
          "type": "card",
          "name": "Daniel Kelly Music",
          "photo": "https://avatars.webmention.io/fsn1.your-objectstorage.com/7aa4815cbc1f993c0e2a6df03280dc168f5ad07fecd0313ddfc27eeb02e0b437.png",
          "url": "https://aus.social/@yasslad"
        },
        "url": "https://mendeddrum.org/@pdcawley/115160370354067277#favorited-by-109307830089461078",
        "published": null,
        "wm-received": "2025-09-07T01:31:46Z",
        "wm-id": 1936817,
        "wm-source": "https://brid.gy/like/mastodon/@pdcawley@mendeddrum.org/115160370354067277/109307830089461078",
        "wm-target": "https://bofh.org.uk/note/7/",
        "wm-protocol": "webmention",
        "like-of": "https://bofh.org.uk/note/7/",
        "wm-property": "like-of",
        "wm-private": false
      },
      {
        "type": "entry",
        "author": {
          "type": "card",
          "name": "Jess Robinson",
          "photo": "https://avatars.webmention.io/fsn1.your-objectstorage.com/a11ee0a58e54873140bf1f1965900378d866fce3fafae79e116b979bea3a8773.jpg",
          "url": "https://fosstodon.org/@castaway"
        },
        "url": "https://mendeddrum.org/@pdcawley/115160370354067277#favorited-by-109562941096076318",
        "published": null,
        "wm-received": "2025-09-07T07:19:01Z",
        "wm-id": 1936873,
        "wm-source": "https://brid.gy/like/mastodon/@pdcawley@mendeddrum.org/115160370354067277/109562941096076318",
        "wm-target": "https://bofh.org.uk/note/7/",
        "wm-protocol": "webmention",
        "like-of": "https://bofh.org.uk/note/7/",
        "wm-property": "like-of",
        "wm-private": false
      }
    ],
    "in-reply-to": [],
    "mention-of": [],
    "repost-of": [],
    "other": []
  }
}

Just imagine that JSON object has a bunch more paths as keys referencing similar objects keyed by mention type.

As it stands, wm--fetch-all is returning a flat sequence of webmention objects that we want to process into a more structured object,

JSON/Javascript calls them objects, old Perl heads like me think of them as hashes, and they’re “Hash Tables” in Emacs Lisp. I’ll be calling them hashes from now on.

in other words we want to fold (or “reduce” in Emacs Lisp terminology) the sequence into a hash. And I know just the function for that. Let’s see what describe-function has to say about seq-reduce:

seq-reduce is a byte-compiled function defined in seq.el.gz.

*Signature*
(seq-reduce FUNCTION SEQUENCE INITIAL-VALUE)

*Documentation*

Reduce the function FUNCTION across SEQUENCE, starting
with INITIAL-VALUE.

Return the result of calling FUNCTION with INITIAL-VALUE
and the first element of SEQUENCE, then calling FUNCTION
with that result and the second element of SEQUENCE,
then with that result and the third element of SEQUENCE,
etc. FUNCTION will be called with INITIAL-VALUE (and then
the accumulated value) as the first argument, and the
elements from SEQUENCE as the second argument.

If SEQUENCE is empty, return INITIAL-VALUE and FUNCTION
is not called.

This does not modify SEQUENCE.

So we can write

(seq-reduce #'wm--add-mention-to-hash-table
            (wm--fetch-all)
            (make-hash-table :test 'equal))

and that will handle the business of iterating over the sequence of mentions for us, and all we have to do is write wm--add-mention-to-hash-table to populate the hash we made with (make-hash-table :test 'equal)

We need to use that :test ’equal part because json-insert wants a hash with strings as keys and the default hash returned by (make-hash-table) compares keys using eql which might or might not work when comparing strings. Not a problem which equal has.

one mention at a time, and return the modified hash (You and I both know that it’s the same old hash mutated, but let’s pretend it isn’t, eh?).

What does that function look like? Here’s what I wrote:

(defun wm--add-mention-to-hash-table (acc mention)
  "Helps reduce a list of mentions into a two level hash."
  (require 'dash)
  (let* ((path (--> mention
                    (gethash "wm-target" it)
                    (url-generic-parse-url it)
                    (url-path-and-query it)
                    (car it)))
         (mentions-hash (or (gethash path acc nil)
                            (wm-new-mentions-hash)))
         (mention-type (gethash "wm-property" mention))
         (mentions (or (gethash mention-type mentions-hash)
                       (progn
                         (setq mention-type "other")
                         (gethash mention-type mentions-hash))))
         (new-mentions (if (seq-contains mentions mention)
                           mentions
                         (vconcat mentions (list mention)))))
    (puthash mention-type new-mentions mentions-hash)
    (puthash path mentions-hash acc)
    acc))

We grab the path from the "wm-target" key, which is actually a URL rather than a simple path

We could just use the URL, and that would work fine on this site, but not when I’m running on localhost. The path will always match with .RelPermalink, but the host part of .Permalink is different in development than in production.

so, rather than writing

(car
 (url-path-and-query
  (url-generic-parse-url
   (gethash "wm-target" mention))))

we’ll thread mention through that series of transformations using dash.el’s threading macro, -->.

We use the path to grab mentions-hash from the acc-umulating hash and, if there isn’t already one there, we grab an empty, but structured hash using wm-new-mentions-hash, which looks like this:

(defun wm-new-mentions-hash ()
  "Make a new empty hash to hold categorised webmention data."
  (copy-hash-table
   #s(hash-table
      test equal
      data ("like-of" [] "in-reply-to" []
            "mention-of" [] "repost-of" []
            "other" []))))

Now we look up "wm-property" in mention, and use that to grab its associated vector of mentions. Well, we would, but there’s a small wrinkle.

We’re only currently interested in four kinds of mention, but Webmention.io doesn’t know that. We could throw the extras away, but what if we became interested in bookmark-of mentions or whatever somewhere down the road. So let’s collect them under the other key. Which is where this hacky section of our let* form comes in:

(mention-type
 (gethash "wm-property" mention))
(mentions
 (or (gethash mention-type mentions-hash)
     (progn
       (setq mention-type "other")
       (gethash mention-type mentions-hash))))

What’s going on here then?

First, we make a guess at the mention-type we’re going to file the current mention under by grabbing the "wm-property" and use that value to lookup the mention type in mentions-hash. If it’s one of the four types we’re interested in, that will be a vector, which is truth-y, otherwise we get nil, which is false-y so we change the mention type to “other” and grab that vector from the mention hash.

We now know the key path we’re going to store our mentions in, and we have the current vector of mentions associated with it. So, if we already know (seq-contains mentions mention) about the current mention, we reuse that, otherwise we make a new vector with the current mention added to it.

That done, it’s a simple matter of putting the new mentions vector into our mentions hash, putting the mentions hash into our accumulating hash and returning that.

With that done, it’s a simple matter of opening data/mentions/all.json, erasing the buffer, calling (json-insert (seq-reduce ...)) to update the data and saving it. Here’s the code which does exactly that.

(defun wm-unflatten-mentions (mentions-vec)
  (seq-reduce 'wm--add-mention-to-hash-table mentions-vec
              (make-hash-table :test 'equal)))

(defun wm-fetch-mentions ()
  "Fetch the webmentions of `wm-domain'."
  (interactive)
  (save-current-buffer
    (let ((all-entries (wm--fetch-all)))
      (with-temp-file (expand-file-name "all.json" wm-data-dir)
        (erase-buffer)
        (json-insert (wm-unflatten-mentions all-entries))))))

Over in the Hugo partial that renders the bit of the page immediately after this, we can get at the data like this:

{{- $all_mentions := index site.Data.mentions.all .RelPermalink  -}}
{{ $likes := index $all_mentions "like-of" | default slice -}}
{{ $reposts := index $all_mentions "repost-of" | default slice -}}
{{ $replies := index $all_mentions "in-reply-to" | default slice -}}
{{ $mentions := index $all_mentions "mention-of" | default slice -}}
<footer class="metaline">
  <ul class="response-summary">
    <li>{{ $likes | len }} {{ if eq 1 (len $likes) }}like{{ else }}likes{{ end }}</li>
    <li>{{ $reposts | len }} {{ if eq 1 (len $reposts) }}repost{{ else }}reposts{{ end }}</li>
    <li>{{ $replies | len }} {{ if eq 1 (len $replies) }}reply{{ else }}replies{{ end }}</li>
    <li>{{ $mentions | len }} {{ if eq 1 (len $mentions) }}mention{{ else }}mentions{{ end }}</li>
  </ul>
</footer>
...

I’ll leave the rest as an exercise for the interested reader. However, I will note that the Webmention.io API includes the option to pass in a since argument, so it wouldn’t be hard to write

(seq-reduce
 'wm--add-mention-to-hash-table
 (wm-fetch-mentions-since wm-last-checked)
 (wm-parse-mentions-file
  (expand-file-name "mentions/all.json"
                    wm-data-dir)))

without having to change our reducing function at all.

Separation of concerns, baby! Separation of concerns!

Tue, 9nd Sep, 2025 8:49pm +0000