Support Links in Jupyter Notebooks?

[mre]

The Jupyiter notebook file format supports Markdown cells, which can contain links. Currently, we extract links using our plaintext extractor, which can lead to false-positives. For instance, see the discussion here: #1658.

There is a crate, nbformat, which would allow us to extract the Markdown cells from a Jupyiter file (.ipynb). This way, we could use a proper Markdown parser for link extraction.

If anyone wants to contribute, take a look at the markdown extractor. The new "notebook extractor" would look quite similar. It would use nbformat to get all notebook cells and then call the Markdown extractor to find all links.

Help wanted! Comment here if you want to give it a shot or send in a pull request. ✌

[duboism]

Quick comment about Jupyter notebooks (as the OP of #1658). In my case, some of the problems didn't come from Mardown cells but from code cells that output URLs (in which case, the URL is written in the JSON format but new lines can be added).

I created a very simple example of the problems we had in our repository: https://colab.research.google.com/gist/duboism/56234c74f2e8de686193f8a3655fc3a4/issue-1669.ipynb. If you open the file as a JSON document you will understand the different case. Note that in both cases adding a dot at the end of the sentence would probably avoid the problem (notebooks are often insufficiently formatted).

[duboism]

In my opinion, it's not necessary to check links in cells' output. This might simplify the work on this.

[mre]

For future reference, here's the JSON version of the linked notebook:

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "source": [
        "Suppose we write a URL in a Markdown cell like https://lychee.cli.rs/\n",
        "\n",
        "This will incorrecly add `/n` at the end of the URL"
      ],
      "metadata": {
        "id": "sLXdkWAj9Q-o"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "QcWMYgcD9PeG",
        "outputId": "ff29856d-c705-40de-aa64-fc45dd4c5f75"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "The best URL link checker of the planet is found at https://lychee.cli.rs/\n"
          ]
        }
      ],
      "source": [
        "# Now let's illustrate a faulty code cell\n",
        "# The URL should be extracted correctly from the code but not from the output (same /n added)\n",
        "print(\"The best URL link checker of the planet is found at https://lychee.cli.rs/\")"
      ]
    }
  ]
}

[mre]

I wrote a small test script for parsing the file.
https://github.com/lycheeverse/notebook

Turns out, version 4.0 is currently not supported.
I get this error when I try it on your file:

Error: UnsupportedVersion(4, 0)

I'm not clear on what the difference is between 4.0 and 4.1 (the first format that is supported. The check is here:
https://github.com/runtimed/runtimed/blob/dcb60f45a414686b59e711a48d451aaf3fec4b95/crates/nbformat/src/lib.rs#L28-L34

@rgbkrk would you be able to assist here? 😅

[duboism]

Another possibility would be to convert the notebook to markdown and then check the links as markdown. There is a jupyter tool to do this called nbconvert. I'm not sure you can use it directly given that it's in python but you may take inspiration from the code.

[mre]

Thanks for the idea. The tool is quite extensive. I wouldn't want to call into Python either. We only need a subset of the functionality. Let's see if nbformat works for us. Perhaps we could extend the support to the 4.0 format.

[iamashwin99]

is it possible to just mention a pattern to ignore at the end?

[mre]

You mean removing the /n at the end of the link?
You can use a remap rule.

The following would change all URLs that end with /n to versions without /n:

lychee --remap '(.*)/n $1'

[iamashwin99]

You mean removing the /n at the end of the link? You can use a remap rule.

The following would change all URLs that end with /n to versions without /n:

lychee --remap '(.*)/n $1'

Perfect, this works for me
I added:

# Remap URLs ending with /n or **/n to the version without the suffix
remap = ["(.*)\\*\\*/n$ $1", "(.*)/n $1", ]

to the lychee.toml which then handles the new line and bold characters that are accidentally picked up from the notebook.