The latest updates on your projects. Learn more about Vercel for Git ↗︎

Ok, so this is a "middle Process" hook along the lines of Preprocess and Postprocess? I like that. Does this also give access to the tokens.links property so we can manipulate the reflink definitions? For example to inject a common link that can be referenced globally without being defined within the markdown input?

Yes it would. It is passed the tokens object that is returned by the lexer after all tokens have been created and before walkTokens functions are called.

Adding to this then: is there a benefit to splitting into two hooks? one immediately after block tokens, and another immediately after inline tokens? I'm thinking of a case where you might want to alter the block-level tokens before their internal tokens get lexed.

A hook that captures block tokens before creating the inline tokens will need to take a bit more thought because of the way we do async marked.

The hook will have to be in the Lexer.lex method and have to return a promise if options.async is true but it currently doesn't return a promise. That might have to be a breaking change.

I think it could be done in another PR.

@calculuschild should I rename this to allTokens so we can add blockTokens later?

Yeah, probably a good idea. I might suggest we use a verb like we have with the other steps, i.e. "preprocess", "walktokens", "parse", etc.

• Preprocess
• Lex
• BlockTokens????
• AllTokens????
• WalkTokens
• Parse
• Postprocess

Maybe "ProcessTokens" or something?

Or like, ProcessBlockTokens, ProcessAllTokens.

Something like that?

Ya I think I like processAllTokens then we can do processBlockTokens later

For reference, this also solves #2639 and proves mind communication.