Syntax highlighting with javascript


I have a very complicated thing to tokenize and grammars would be very painful to do. Is there a way to add the CSS classes with JavaScript instead of grammar files?

EDIT: Basically, how & when does atom add the classes, and how should I do that


If a file is loaded without a grammar, every line will be an uninterrupted string inside its own div. So, yes.

I find it difficult to understand why a grammar would be more complicated than hard-coding the exact same thing in JavaScript. The hard work has already been done and you’re proposing to re-do it.

Basically, how & when does atom add the classes

When it reads the grammar and uses that to determine what to tokenize.


How would I actually add the classes, and when should the event happen


If you don’t know JavaScript well enough to add classes to a block of text, you definitely shouldn’t build your own tokenizer.


I do know it that well, but I’m wondering what is the correct way of doing it in atom, and how would I trigger the event


There isn’t a correct way to do it in Atom, because Atom has a grammar system that takes care of it. You’d have to get into direct DOM manipulation. Like I said, it’s not easier than writing a grammar.

You’d want to watch editor.onDidChange(), and also figure out how to watch when Atom changes its scroll position.


Also, do you know where atom adds the classes in the source code?


It happens here.


But where in the file?


All of it. And token-iterator.js and tokenized-line.js are also important.


I don’t see anywhere in those files where it actually edits the DOM. Any ideas where that would be?


It doesn’t edit the DOM directly. It generates a tokenized buffer that later on gets rendered here. For things as complicated as Atom’s front end, you can’t be making a bunch of direct edits because you run the risk of running into your own changes. Instead, when an update happens to the information about what you should see (the model), the whole thing is computed and re-drawn. Computers are fast enough these days that, for most files, we don’t notice the difference (but, IIRC, this is part of why Atom chokes on very large files).


So how would I feed atom a tokenized buffer that I created?


My apologies for stepping into a topic I know nothing about.

@MrYurihi, could you explain more about the problem you wish to solve. What is the ‘very complicated thing’ you hope to put syntax highlighting to? Perhaps some grammar wizard has some suggestions to make.

…perhaps working the problem from two ends.

Good luck.


The problem is that each part of a line is dependent on the first word in the line, and depending on the word, it changes how to highlight the thing. I already can highlight it well, but I want to have a better way to highlight it, along with have a popup that shows you what you are hovering over. see . Also I’m just interested on how to do this


Maybe you mean:

It is still not clear to me what you are after. If you are looking for the popup, that can be done via autocomplete’s API. Highlighting depending on preceding words are handled by the grammar. I am not sure you could manipulate the tokenizer to do what you seek… it might even be the wrong place to look for an answer.

Please be patient with me - spell it out what result you are after. The picture in your package does not tell me anything. An example would be nice.


I just want to know how to feed atom a tokenized buffer that I created


I know how to do that, but I want to create my own tokenized buffer, then feed it to atom so I can display that.


I’m not sure how to convince you that it’s not easier to rewrite a good chunk of Atom’s internals than it is to create a language grammar. I’m also not going to do all of your work for you. How do you expect to edit Atom’s internals if you have to have someone explain them to you?


I don’t know why you believe that this is too complicated for a grammar. It’s not.