Coding Adventure: Micro.blog Bookmarks & Highlights

Life has been way too much lately and most of my hobbies were put on pause while I dealt with keeping up with the unexpected. One of those that was dropped was writing down weekly notes that focused on my development side projects (and honestly, even coding side-projects took a nose-dive). My last post was on March 1st 😱

So I’m doing a little reframing to make it feel a bit easier to write, thus the “Coding Adventure”. I want to keep these posts more along the line of “Hey, here’s something fun I played around with and it might still be mostly broken 🤪”

Background

I use Micro.blog, I know…. you’re shocked 😝 and one of the things I love, is how open they are with their API. It’s a fun playground of potential side-projects just waiting to be explored. One piece that I’ve dabbled with is the bookmarks feature. Basically you can save URLs as bookmarks and if you are a premium member, organize them with tags and add highlights. Manton, the creator of Micro.blog, rolled out a feature to bookmarks including having AI summarize the link (only if you’ve enabled AI on your account and, important to me, if the robots.txt of the bookmarked link allows it).

But it’s not a feature I use all that much and I wanted to see if I could change that with a few of my own touches.

Getting Started

Like most of my recent side-projects, I’ve been using Deno. Its quick to get started with a Github repo and they have hard free tier limits so I feel safe playing around.

The first thing I needed was a valid authorization token from micro.blog. I got this using indieauth and saving the resulting token in a http-only cookie. That way I’ll have the token with each page request. To make things easier in this proof of concept, I hard-coded values inside the server generated HTML form I was using. For instance the form action, the scope and response type. On the server I generated a random string for the state (the ${} notation is a JavaScript template literal) and I also included the state as a session cookie when the page is rendered.

<form action="https://micro.blog/indieauth/auth" method="get">
    <input type="url" placeholder="https://username.micro.blog" name="me">
    <input type="hidden" name="client_id" value="https://my-demo.com"/>
    <input type="hidden" name="redirect_uri" value="https://my-demo.com/auth"/>
    <input type="hidden" name="state" value="${uuid}"/>
    <input type="hidden" name="scope" value="create"/>
    <input type="hidden" name="response_type" value="code"/>
    <button type="submit">Sign In</button>
</form>

Submitting this form creates a get request to Mico.blog’s indieauth endpoint which redirects me to approve the login. Once I approve it, it sends a request back to my /auth endpoint with a code and state in the query string of a URL. I check that the state matches the state inside the cookie and then I make a POST request to Micro.blog’s token endpoint, https://micro.blog/indieauth/token, using the code. If everything was wired up correctly I get back my lovey authorization token. Here is a code snippet of the token portion:

const formBody = new URLSearchParams();
formBody.append("code", code);
formBody.append("client_id", 'https://my-demo.com');
formBody.append("grant_type", "authorization_code");

// request access_token
const fetching = await fetch('https://micro.blog/indieauth/token', {
    method: "POST",
    body: formBody.toString(),
    headers: {
        "Content-Type": "application/x-www-form-urlencoded; charset=utf-8",
        "Accept": "application/json"
    }
});    
const response = await fetching.json();

A Fetch Quest: Getting My Bookmarks

Okay, let’s get some bookmarks. Micro.blog has documentation on its bookmarks endpoints in its help center, so that’s the first place to go diving. This was all familiar to me since I put bookmarks in my side project Lillihub a while back.

let fetching = await fetch(`https://micro.blog/posts/bookmarks`, { 
    method: "GET", 
    headers: { "Authorization": "Bearer " + access_token } 
});
let results = await fetching.json();

This returns twenty-five of my most recent bookmarks. Nice. There was some trial and error but I figured out that you can use query parameter before_id to enable getting more bookmarks. For example, if the last bookmark id was 123123 then I could call https://micro.blog/posts/bookmarks?before_id=123123 and get the twenty-five after that last bookmark.

If you have tags you can get the by adding the tag with the name of the tag, like this: https://micro.blog/posts/bookmarks?tag=code and you will get the first 25 bookmarks with that tag.

Lost in the Woods: Bookmarks and What I Couldn’t Figure Out

What I couldn’t figure out is how to go backwards. The Micro.blog post API supports a since_id parameter and a count parameter allowing for a full paging experience. Those don’t seem to work on bookmarks endpoint. I tried grabbing the bookmarks via the micropub endpoint but I had no luck there, just getting the posts on my blog.

Nor could I figure out how to search for a bookmark. I figured this would be handy with the AI summarization or even to search part of the bookmark title. There is limited support for searching on the Micro.blog website but it mixes results together from the other related sections of links and highlights.

So two paths then. Load all the bookmarks on the HTML page and then use JavaScript to search OR load all the bookmarks on the server, search it, and then return the results.

Since I don’t have thousands of bookmarks… yet. I’m going with the quick and easy client side solution.

Search For Treasure: Enabling Search

The while loop was my weapon of choice for this part of the adventure. I iterated calling the bookmarks API and gathered the results. And I kept doing that until the request returned less than 25 results… signaling the end.

Then I put those bookmarks in a table with some styling and….

Nice.

The search box runs a little bit of JavaScript that hides table rows if they don’t contain the search term. Clicking on a tag, whether at the top or on a bookmark fills the textbox and retriggers the search.

One strange thing with the Micro.blog bookmark API is that it doesn’t have separate properties for things like title or reader link. It jumbles them all together in the content_html field.

{
    ...,
    "content_html": "<p>Homemade Oreos <a href="https://sallysbakingaddiction.com/homemade-oreos/">sallysbakingaddiction.com</a></p><p class="post_archived_links">Reader: <a href="https://micro.blog/bookmarks/6888713\">sallysbakingaddiction.com</a> </p>",
    ...
},

So I’m displaying the bookmark title as what’s in the first paragraph tag and removing everything else.

Side Quest: Adding Bookmarks

This was actually something I figured out a while back, but I figured I’d actually list the code sample here for a more complete picture.

First you need to create the bookmark using micropub:

const formBody = new URLSearchParams();
formBody.append("h", "entry");
formBody.append("bookmark-of", url);

const posting = await fetch(`https://micro.blog/micropub`, {
    method: "POST",
    body: formBody.toString(),
    headers: {
        "Content-Type": "application/x-www-form-urlencoded; charset=utf-8",
        "Authorization": "Bearer " + accessTokenValue
            }
});

and then you need to add the tags using the Micro.blog bookmarks API.

1. Get the bookmarks
2. The new one should be first one returned, grab it’s id
3. Add the tags (comma-separated list)

const formBodyTags = new URLSearchParams();
formBodyTags.append("tags", tags);
           
const postingTags = await fetch(`https://micro.blog/posts/bookmarks/${bookmark.id}`, {
    method: "POST",
    body: formBodyTags.toString(),
    headers: {
        "Content-Type": "application/x-www-form-urlencoded; charset=utf-8",
        "Authorization": "Bearer " + accessTokenValue
    }
});
const responseTags = await postingTags.text();

Exploring the Dungeon: Highlights

Let’s circle back to that reader link in the content_html property. Micro.blog premium users can make highlights on bookmarks, assuming the website’s robot.txt doesn’t prohibit it being crawled and the content isn’t dynamic. A copy of the content is provided along with an interface to make highlights. This video at the 2:52 mark by @manton shows the functionality I’m talking about.

First thing I wanted to change, I want the number of highlights displayed in-line with my bookmarks. So I fire off the request to get my highlights and the first problem becomes apparent. There is no id link between a bookmark and a highlight, or at least one that I could spot.

So…. what matches? The url of the bookmark and the url of the highlight. So I can match them up with that. Downside? I’ll need to get all the highlights and loop through to find matches on the bookmarks. That’ll work for my small collection, but probably not for someone with a lot of bookmarks/highlights.

With that I was able to add a little marker to tell me when I have highlights. 🥳… one monster down 🧟‍♀️⚔️

Now I’m getting to the tricky part.

One thing I don’t like with the highlights… is that you can’t highlight the underlying HTML. Which means that if you highlight some text with an anchor tag, you get the text of the anchor, but not the anchor link itself. Which means the handy web reader you can get from Micro.blog won’t work out of the book. First, here is a look at a different part of the JSON returned for a bookmark:

   {
      ...
      "_microblog": {
        "links": [
          {
            "id": 6888713,
            ....,
          }
        ],
      }
    }

You can’t use the id of the bookmark to get the Micro.blog reader. The endpoint is https://micro.blog/hybrid/bookmarks/123 , with the 123 being the id of the first object under the _microblog.links property in a bookmark response. This returns a text response that is an HTML page. Sweet.

Now I can start dissecting it!

In Which I Spend Way Too Much Time Staring At Source Code

This was the tedious part of the adventure. I needed to parse out what was being returned, rip it apart and then rebuild it. I’m on a personal mission to not use any JavaScript libraries, so….

Step one: Bye bye JQuery 👻⚔️

Luckily the contents of the web reader page has a div around it with an id <div id="content">. This means I can get the content of the bookmark. A point in my favor, the inline JQuery script is outside that div. But by removing JQuery the two helpful script tags included by Micro.blog aren’t going to work anymore.

Step two: Fix the broken images 🧪🔨

So if the bookmark had images in it… they are all broken. Why? because where my app is running is not where the images are. Thankfully I have a base url to work with. It was included in the head of the HTML returned by Micro.blog. I grabbed the <base href="" /> extracted the value and then prepended it to all the image src’s.

Step Three: Mark up the existing highlights 🖊️🧝🏻‍♀️

Another downside, using just the reader page I don’t have the highlighted text… or title. Micro.blog seems, I’m not sure on this since I can’t see their code, to generate the highlight inside a script, marking the start and end range of the highlight… like this restoreHighlight(0, 1132, 1225); with restoreHighlight being a custom function included on the page.

For now I’m to loop through my highlights and find the ones I need. But this a place where I may need to rethink my approach. Now that have the text of the highlight, I loop through each one and add a <mark> tag around the matching text of the content. This very neatly adds a highlight, no additional CSS needed.

Step Four: Write new JavaScript to capture highlighted text 🐲🔥

Note: I’ve only tested this on browsers I use…

Okay, let’s get technical. I need to listen for a text selection event and then get what was selected. Plus I really only need to care if something was actually selected.

document.addEventListener("selectionchange", function(event) {
    const selection = window.getSelection();     
    if (selection.rangeCount === 0) {
        return;
    }
    getSelectionHtml();
});

the getSelectionHtml() function I wrote does all the nitty gritty calculating and saves the information in a hidden form on the page. This form appears in a dialog modal after a one second delay. If the submit button is hit, it posts the form and I save the highlight. This Micro.blog endpoint wasn’t actually included in the documentation… but it works:

const form = new URLSearchParams();
form.append("text", text);
form.append("start", start);
form.append("end", end);
         
const posting = await fetch(`https://micro.blog/bookmarks/${id}/highlights`, {
    method: "POST",
    body: form.toString(),
    headers: {
        "Content-Type": "application/x-www-form-urlencoded; charset=utf-8",
        "Authorization": "Bearer " + mbToken
    }
});

Note, the id is the id of the reader, not the bookmark.

Okay, so the getSelectionHtml() is a bit of a beast. I haven’t really played around with widow.getSelection() before and I wanted to remove the JQuery dependency with the original code. Luckily it didn’t need too much tweaking to make it work.

First we want to make sure our selection isn’t undefined. Then we want to create a container (in this case a div) to hold our HTML selection. I then iterate through and clone the nodes into that container. Then I can get the innerHTML of the container and that holds the user selected HTML. For brevity I took out the calculateOffset portion and left comments where that code would go.

function getSelectionHtml() {
    var html = "";
    if (typeof window.getSelection != "undefined") {
        var sel = window.getSelection();
        if (sel.rangeCount) {
            var container = document.createElement("div");
            for (var i = 0, len = sel.rangeCount; i < len; ++i) {
                container.appendChild(sel.getRangeAt(i).cloneContents());
            }
            html = container.innerHTML;
            if (!sel.isCollapsed) {
                //save the HTML to the form
                //calculate the offsets and save values to form
                //trigger the 1 second delay
            }
        }
    } else if (typeof document.selection != "undefined") {
        if (document.selection.type == "Text") {
            html = document.selection.createRange().htmlText;
            //save the HTML to the form
            //calculate the offsets and save values to form
            //trigger the 1 second delay
        }
    }
}

To calculate the offset you need to gather up all the child nodes. Luckily this code was already included in the page. I just needed to remove the JQuery portion and port it over to vanillaJS.

function gatherNodes(startElement, results) {
    results.push(startElement);
    startElement.childNodes.forEach(function (e) {
        gatherNodes(e, results);
    });
}
                  
function calculateContentOffset(findNode, offset) {
    var e = document.getElementById("content");
    var results = Array();
    gatherNodes(e, results);
    var current_pos = 0;
    for (var i = 0; i < results.length; i++) {
        var node = results[i];
        if (node.isSameNode(findNode)) {
            return current_pos + offset;
        }
        if (node.length != undefined) {
            current_pos = current_pos + node.length;
        }
    }
}

Whew….

It works!

And if I double check it on Micro.blog, I can see the link is saved. Yep!

Or maybe not…. 😱 As you can see, there is a mismatch between what’s in my UI and on the Micro.blog website. I wonder if the offset I’m calculating is different somehow… 🤔

There Are Monsters Yet To Slay🦇

1. Highlights saved with my app that have HTML in them… aren’t visible or in the wrong position in the Micro.blog reader on the Micro.blog website. They do show up correctly in the highlight list and when hitting the “New Post” button. Which was one of the reasons I started this adventure.
2. Highlights saved with the Micro.blog website that cover HTML elements (like anchors) don’t show in my reader.

So this isn’t ready for prime-time.

Conclusion

I now have my own little web reader for my bookmarks and I can mark up highlights. It has its drawbacks, but I’m going to use it for a bit and see how it holds up.

I think that’s all the adventuring I’m up for this week. I’m going home and having a nice cup of peppermint tea.