Customize your own Markdown compiler?

Here’s how you can use mistletoe in a Python script:

import mistletoe

with open('foo.md', 'r') as fin:
    rendered = mistletoe.markdown(fin)Copy the code

mistletoe.markdown() uses mistletoe’s default settings: allowing HTML mixins and rendering to HTML. The function also accepts an additional argument renderer. To produce LaTeX output:

import mistletoe
from mistletoe.latex_renderer import LaTeXRenderer

with open('foo.md', 'r') as fin:
    rendered = mistletoe.markdown(fin, LaTeXRenderer)Copy the code

Finally, here’s how you would manually specify extra tokens and a renderer for mistletoe. In the following example, we use HTMLRenderer to render the AST, which adds HTMLBlock and HTMLSpan to the normal parsing process.

from mistletoe import Document, HTMLRenderer

with open('foo.md', 'r') as fin:
    with HTMLRenderer() as renderer:
        rendered = renderer.render(Document(fin))Copy the code

mistletoe is the fastest Markdown parser implementation available in pure Python; that is, on par with mistune. Try the benchmarks yourself by running:

python3 test/benchmark.pyCopy the code

One of the significant bottlenecks of mistletoe compared to mistune, however, is the function overhead. Because, unlike mistune, mistletoe chooses to split functionality into modules, function lookups can take significantly longer than mistune.

To boost the performance further, it is suggested to use PyPy with mistletoe. Benchmark results show that on PyPy, mistletoe is about twice as fast as mistune:

$ pypy3 test/benchmark.py mistune mistletoe Test document: test/samples/syntax.md Test iterations: 1000 Running tests with mistune, mistletoe... = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = mistune: mistletoe 13.524028996936977:6.477352762129158Copy the code

The above result was achieved on PyPy 5.8.0-beta0, on a 13-inch Retina MacBook Pro (Early 2015).

GitHub wiki links are span-level tokens, meaning that they reside inline, and don’t really look like chunky paragraphs. To write a new span-level token, all we need to do is make a subclass of SpanToken:

from mistletoe.span_token import SpanToken

class GithubWiki(SpanToken):
    passCopy the code

mistletoe uses regular expressions to search for span-level tokens in the parsing process. As a refresher, GitHub wiki looks something like this: [[alternative text | target]]. We define a class variable, pattern, that stores the compiled regex:

class GithubWiki(SpanToken):
    pattern = re.compile(r"\[\[ *(.+?) *\| *(.+?) *\]\]")
    def __init__(self, match_obj):
        passCopy the code

For spiritual guidance on regexes, refer to xkcd classics. For an actual representation of this author parsing Markdown with regexes, refer to this brilliant meme by Greg Hendershott.

mistletoe’s span-level tokenizer will search for our pattern. When it finds a match, it will pass in the match object as argument into our constructor. We have defined our regex so that the first match group is the alternative text, and the second one is the link target.

Note that alternative text can also contain other span-level tokens. For example, [[*alt*|link]] is a GitHub link with an Emphasis token as its child. To parse child tokens, simply pass match_obj to the super constructor (which assumes children to be in match_obj.group(1)), and save off all the additional attributes we need:

from mistletoe.span_token import SpanToken

class GithubWiki(SpanToken):
    pattern = re.compile(r"\[\[ *(.+?) *\| *(.+?) *\]\]")
    def __init__(self, match_obj):
        super().__init__(match_obj)
        self.target = match_obj.group(2)Copy the code

There you go: a new token in 7 lines of code.

It is preferred that all mistletoe’s renderers be used as context managers. This is to ensure that your custom tokens are cleaned up properly, so that you can parse other Markdown documents with different token types in the same program.

from mistletoe import Document
from plugins.github_wiki import GithubWikiRenderer

with open('foo.md', 'r') as fin:
    with GithubWikiRenderer() as renderer:
        rendered = renderer.render(Document(fin))Copy the code

For more info, take a look at the base_renderer module in mistletoe. The docstrings might give you a more granular idea of customizing mistletoe to your needs.

• I am interested in a Markdown-to-LaTeX transpiler in Python.
• I want to write more Python. Specifically, I want to try out some bleeding edge features in Python 3.6, which, in turn, makes me love the language even more.
• I am stuck at home during summer vacation without an internship, which, in turn, makes me realize how much I love banging out software from scratch, all by myself. Also, global warming keeps me indoors.
• I have long wanted to write a static site generator, from scratch, by myself. One key piece of the puzzle is my own Markdown parser. “How hard could it be?” (well, quite a lot harder than I expected.)
• “For fun,” says David Beasley.

Here’s two things mistune inspired mistletoe to do:

• Markdown parsers should be fast, and other parser implementations in Python leaves much to be desired.
• A parser implementation for Markdown does not need to restrict itself to one flavor (or, “standard”) of Markdown.

Here’s two things mistletoe does differently from mistune:

• Per its readme, mistune will always be a single-file script. mistletoe breaks its functionality into modules.
• mistune, as of now, can only render Markdown into HTML. It is relatively trivial to write a new renderer for mistletoe.
  • This might make mistletoe look a bit closer to MobileDoc, in that it gives simple Markdown additional power to deal with a variety of additional input and output demands.

The implications of these are quite profound, and there’s no definite this-is-better-than-that answer. Mistune is near perfect if one wants what it provides: I have used mistune extensively in the past, and had a great experience. If you want more control, however, give mistletoe a try.

Finally, to quote Raymond Hettinger:

If you make something successful, you don’t have to make something else unsuccessful.

Messing around in Python and rebuilding tools that I personally use and love is an immensely more rewarding experience than competition.