Álvaro Ramírez

A recent Hacker News post, Ask HN: Favorite Blogs by Individuals, led me to dust off my oldie but trusty command to extract comment links. I use it to dissect these wonderful references more effectively.

You see, I wrote this command back in 2015. We can likely revisit and improve. The enlive package continues to do a fine job fetching, parsing, and querying HTML. Let's improve my code instead… we can shed a few redundant bits and maybe use newer libraries and features.

Most importantly, let's improve the user experience by sanitizing and filtering URLs a little better.

We start by writing a function that looks for a URL in the clipboard and subsequently fetches, parses, and extracts all links found in the target page.

(require 'enlive)
(require 'seq)

(defun ar/scrape-links-from-clipboard-url ()
  "Scrape links from clipboard URL and return as a list. Fails if no URL in clipboard."
  (unless (string-prefix-p "http" (current-kill 0))
    (user-error "no URL in clipboard"))
  (thread-last (enlive-query-all (enlive-fetch (current-kill 0)) [a])
    (mapcar (lambda (element)
              (string-remove-suffix "/" (enlive-attr element 'href))))
    (seq-filter (lambda (link)
                  (string-prefix-p "http" link)))
    (seq-uniq)
    (seq-sort (lambda (l1 l2)
                (string-lessp (replace-regexp-in-string "^http\\(s\\)*://" "" l1)
                              (replace-regexp-in-string "^http\\(s\\)*://" "" l2))))))

Let's chat (current-kill 0) for a sec. No improvement from my previous usage, but let's just say building interactive commands that work with your current clipboard (or kill ring in Emacs terminology) is super handy (see clone git repo from clipboard).

Moving on to sanitizing and filtering URLs… Links often have trailing slashes. Let's flush them. string-remove-suffix to the rescue. This and other handy string-manipulating functions are built into Emacs since 24.4 as part of subr-x.el.

Next, we can keep http(s) links and ditch everything else. The end-goal is to extract links posted by users, so these are typically fully qualified external URLs. seq-filter steps up to the task, included in Emacs since 25.1 as part of the seq.el family. We remove duplicate links using seq-uniq and sort them via seq-sort. All part of the same package.

When sorting, we could straight up use seq-sort and string-lessp and nothing else, but it would separate http and https links. Let's not do that, so we drop http(s) prior to comparing strings in seq-sort's predicate. replace-regexp-in-string does the job here, but if you'd like to skip regular expressions, string-remove-prefix works just as well.

Yay, sorting no longer cares about http vs https:

https://andymatuschak.org
http://antirez.com
https://apenwarr.ca/log
...

With all that in mind, let's flatten list processing using thread-last. This isn't strictly necessary, but since this is the 2021 edition, we'll throw in this macro added to Emacs in 2016 as part of 25.1. Arthur Malabarba has a great post on thread-last.

Now that we've built out ar/scrape-links-from-clipboard-url function, let's make its content consumable!