How does Atom handle grammar highlighting?


#1

Could anyone with a working knowledge of Atom’s internals give me some pointers on how exactly does Atom turn token list received from grammar into span elements? I’ve tried going over Atom’s sources, but wasn’t able to find the exact place.

There are two reasons why I’m asking this.

First one is I want to do the same in my package, and was wondering if I could just reuse Atom’s implementation instead of rolling my own.

Second is I’m thinking about running some code on EditorElement right after token spans are created, so I was wondering if at all possible to do that synchronously.


#2

tokenized-buffer-iterator.coffee is the code that does the tag-adding.


#3

Thanks. So it seems like for (2) my only options are monkey-patching TextEditorPresenter.updateLines or TextEditorComponent.updateSync. Not ideal.

As for (1), apparently I will have to basically create a TextEditor and work from that. Huh. Rolling my own suddenly doesn’t sound like that bad of an idea…


#4

For (1), what exactly do you plan to do that you couldn’t do by making a custom grammar? For (2), what changes between the <span> tags being added and the finished state that gets in the way of what you want?


#5
  1. I want to highlight contents of a div. Pretty straightforward, yes? It just occured to me that I could probably use https://github.com/atom/highlights though.
  2. Lack of events? I.e. I need to monkey-patch a hook in to actually run something synchronously right after highlighter’s span nodes are created. Or am I missing something? I.e. I can run that code asynchronously, but that doesn’t look too good. Since I’m aiming to manipulate span nodes directly to change display style (more specifically, add some css classes to some nodes), running that asynchronously can lead to flicker.

#6

It’s so straightforward that I’m still confused. One of the easiest things to do with JS is manipulate style rules, and Atom includes an API for decorations that fall outside the scope of the grammar highlighting.

Lack of events? I.e. I need to monkey-patch a hook in to actually run something synchronously right after highlighter’s span nodes are created.

And you’ve checked out the did-tokenize event and it doesn’t work?


#7

Sorry about being confusing. Point is, I have a string that contains some code, and I have a grammar for it. Now I want to apply latter to former and put result into a custom panel (well, basically a div). It’s relatively straightforward to do just by iterating over tokens, but code gets pretty gnarly, so I was wondering if it’s possible to reuse Atom’s implementation. Apparently that’s not that simple.

For one, it doesn’t seem to fire every time, e.g. it never seems to fire when retokenizing a part of buffer (as a result of line change etc), although it does fire on grammar change. I’m not sure if it’s a bug or a feature.
EDIT: looking at the code more closely, it seems that did-tokenize fires only when tokenizer runs in background. When invalidated chunk is sufficiently small, it doesn’t.


#8

So you essentially want to make a custom panel behave like a TextBuffer with regard to code highlighting? If you don’t need the other features of a tokenized buffer, it could just be simpler to use an external highlighting solution like Highlight.js.

did-invalidate-range appears to happen when small areas are invalidated (so the whole buffer doesn’t need to be tokenized).


#9

But it feels kinda silly, given that everything seems to be already in place in Atom.[quote=“DamnedScholar, post:8, topic:32660”]
did-invalidate-range appears to happen when small areas are invalidated (so the whole buffer doesn’t need to be tokenized).
[/quote]

Nope. tokenizeNextChunk isn’t ever called if chunkSize is bigger than invalidated range, so no events are fired. See bufferDidChange and buildTokenizedLinesForRows.


#10

It’s not silly if it’s easier. Atom’s tokenizer and the features that depend on it could have been broken apart and modular with events at every intersection, but they’re not. Highlight.js is tiny and would take no time at all to implement.


#11

Except I would have to write a grammar for HighlightJS. Not something I’m too keen on doing, considering I already wrote one for Atom. I think we can drop this particular question at this point. I’m just going to take code from Atom’s highlights npm package and rip out anything I don’t like (loading from fs, etc) That way, I can get pretty much what I want in 80 lines of code or less.