Did displayBuffer.tokenizedBuffer.tokenizedLines change recently?


I have been building the https://github.com/irrationalistic/atom-tasks package and recently encountered a bug that wasn’t present before. Whenever I use displayBuffer.tokenizedBuffer.tokenizedLines I tend to get inconsistent behavior. In some cases it will show all tokens at all depths for the given line. In other cases it will only show the top level items, not the children of those tokens. Is there a more accurate readout somewhere of all tokens on a given line? Or is this a bug? Any suggestions would be welcome! Thanks!


There’s been a ton of performance work that has been done to optimize the scrolling speed of large files. I wouldn’t be surprised if some of this work has changed the stuff you’re talking about. On the other hand, since both DisplayBuffer and TokenizedBuffer aren’t public APIs, this can’t really be considered a “bug”.

I’ll see if I can find the PRs that might have touched this, or you can look for PRs by @as_cii or @nathansobo.


Thanks! Sorry, meant a bug with my plugin :slight_smile:

Totally understand about the performance work. I can probably find a workaround if need-be. I’ll keep this thread updated as I find out more in case anyone is interested!


Not entirely sure what caused the actual issue, but it came down to my plugin relying on functionality in TokenizedBuffer, which allowed me to hook into the post-processed work the editor did when parsing the file through the grammar. To bypass this I updated the plugin to run that process manually in my own code. Meant a bunch of reparsing, but no more dependency!

Tests also happened to break because they were using tokenized lines AND the now deprecated /deep/ selector. Sigh. That’s another problem for later though! (Unless anyone has suggestions).

The code is a bit rushed, so it needs cleanup, but here’s the commit for reference: