Trying to make a start with Tree-Sitter; and failing at it


#1

Hello.

The objective is to try and experiment with Tree-Sitter to construct a grammar for Atom. I am following the directions given in https://tree-sitter.github.io/tree-sitter/creating-parsers. Windows 10 Prof. O/S is installed on my machine.

I am already stuck with…

Add tree-sitter-cli to the devDependencies section of package.json and run the command npm install.

Firstly I did not have node installed. That was easy to resolve as this was specified in the write-up (see link).


Then running npm install for first time. The error came up with node-gyp complaining about the installed Python version. It wants Python V2.7. That was unexpected and not even mentioned in the write-up. I installed Python V2.7 and placed it in the PATH.


Next there was a complaint about nan. Not sure how to resolve this one. I took a change and look at the listing in https://github.com/Aerijo/tree-sitter-biber/blob/28ca54d63729b9d621bace539a6ae7eee2083f81/package.json#L21. (ring @Aerijo)


npm install now gives feedback that ends as:

Click to see detail of command line error

… command was executed in (elevated) command prompt
… with admin rights and firewall disconnected.

Error: Cannot find module 'nan'
    at Function.Module._resolveFilename (module.js:547:15)
    at Function.Module._load (module.js:474:25)
    at Module.require (module.js:596:17)
    at require (internal/module.js:11:18)
    at [eval]:1:1
    at ContextifyScript.Script.runInThisContext (vm.js:50:33)
    at Object.runInThisContext (vm.js:139:38)
    at Object.<anonymous> ([eval]-wrapper:6:22)
    at Module._compile (module.js:652:30)
    at evalScript (bootstrap_node.js:466:27)
gyp: Call to 'node -e "require('nan')"' returned exit status 1 while in binding.gyp. while trying to load binding.gyp
gyp ERR! configure error
gyp ERR! stack Error: `gyp` failed with exit code: 1
gyp ERR! stack     at ChildProcess.onCpExit (C:\Program Files\nodejs\node_modules\npm\node_modules\node-gyp\lib\configure.js:336:16)
gyp ERR! stack     at emitTwo (events.js:126:13)
gyp ERR! stack     at ChildProcess.emit (events.js:214:7)
gyp ERR! stack     at Process.ChildProcess._handle.onexit (internal/child_process.js:198:12)
gyp ERR! System Windows_NT 10.0.17134
gyp ERR! command "C:\\Program Files\\nodejs\\node.exe" "C:\\Program Files\\nodejs\\node_modules\\npm\\node_modules\\node-gyp\\bin\\node-gyp.js" "rebuild"
gyp ERR! cwd D:\zooTree\node_modules\tree-sitter-cli
gyp ERR! node -v v8.11.3
gyp ERR! node-gyp -v v3.6.2
gyp ERR! not ok
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! tree-sitter-cli@0.13.5 install: `node-gyp rebuild`
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the tree-sitter-cli@0.13.5 install script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR!     C:\Users\dp\AppData\Roaming\npm-cache\_logs\2018-08-02T19_36_00_174Z-debug.log

Now I am stuck.

The package.json is as follows:

{
    "name": "zoo-tree",
    "version": "0.0.1",
    "description": "Testing Tree-Sitter grammar definition",
    "keywords": [],
    "dependencies": {
      "nan": "^2.4.0"
    },
    "devDependencies": {
      "tree-sitter": "latest",
      "tree-sitter-cli": "latest"
    },
    "license": "UNLICENSED",
    "private": true
}

The documentation only mentioned tree-sitter-cli as a dependency. The other two is a guess from the MIT licensed code sample.

Anyone with suggestions?
Thank you for your consideration.







Notes:

  • Windows10 Prof.
  • Start with a non-programmer’s PC,
    except Python V3.7 (now removed)
  • Newly installed:
    - Node
    - Python V2.7

#2

Node-gyp requires Python 2. I guess it’s not mentioned in the tree-sitter docs because

  1. python2 is normally installed on a computer,
  2. node-gyp is notorious (to me at least) for using Python 2 (and failing to find it when python points to Python 3), and
  3. the docs are more focused on writing grammars than setting up dependencies.

Anyway, looks like you solved that issue fine.

I don’t actually know what nan does, I just copied the dependencies of tree-sitter-javascript. It seems they’ve added more dependencies, and removed others. Here’s what I was able to compile and test a grammar with just now:

{
  "name": "tree-sitter-dummy",
  "version": "0.0.0",
  "description": "Tree sitter grammar",
  "main": "index.js",
  "repository": null,
  "author": "Benjamin Gray",
  "license": "MIT",
  "devDependencies": {
    "tree-sitter-cli": "^0.13.5"
  },
  "scripts": {
    "build": "tree-sitter generate && node-gyp build"
  }
}
  • EDIT: ^0.12.5 -> ^0.13.5. Official recommendation is to build for Beta, and eventually things will settle down enough to not worry about different versioning.

The scripts field lets you run npm run build to rebuild the grammar after you make changes. Note that the first time you are building a grammar, you should run these commands instead:

  1. tree-sitter generate
  2. node-gyp configure
  3. node-gyp build

If it complains about “tree-sitter not defined” or similar, ensure you’ve added things to your path like described here.

Once you’ve built it, run tree-sitter test to test all files in the corpus directory.

I did not get any complaints about nan. It was installed automatically as a dependency of tree-sitter-cli, but that’s it.

NOTE: The file structure (before compiling the grammar) should be

tree-sitter-dummy
|- corpus
|  |- testFile1.txt
|  |- testFile2.txt
|  `- etc.
|- grammar.js
`- package.json
  • All other files are generated when building.

  • Use the txtt extension for test files if you want my nifty syntax highlighting (tree-sitter only)

I will probably write a proper beginners guide at some point.


#3

@Aerijo: Benjamin - Thank you for your time + attention.

That would be nice.

Not for Windows machines. (Did I mention I am a Windows user? :wink:)

This is what steals the steam from my kettle… Instructions do not keep in mind to describe how a baseline looks like. A description of the prerequisites would be most welcome. Perhaps not being on Windows is one of the prerequisites.




The notes that you mentioning about the scripts description was insightful. So also the notes following. The stuck in the mud is however with nan. How to get this resolved for me to move forward?

Let me mention that I have no C compiler or anything like that. Which tools are required?

Hopefully waiting for some inspiration.


#4

Windows is fine, I meant that a programmer would typically have it installed for the various projects that might need it (a lot might have utility scripts etc). I get that not everyone does, but I didn’t write those docs either.

And I don’t know about the compiling tools. I’ll get back to you on that.


#5

Howzit.

Hopefully you know I am not directing my frustration at you.

Thank you.

It will be very helpful to know what to study up on or what tools to have for the job when beginning at ground level. I’ll even dust off my old turn-of-the century C-language books. :wink:




UPDATE 1:
This seems to be a problem I have seen before. :blush:

Looking at https://www.npmjs.com/package/node-gyp the following:

On Windows

Option 1

Install all the required tools and configurations using Microsoft’s windows-build-tools using npm install --global --production windows-build-tools from an elevated PowerShell or CMD.exe (run as Administrator).

I’ll be testing that at some point.




UPDATE 2:

Tasks completed (failed result)

  1. npm install --global --production windows-build-tools
    (elevated Poweshell)

  2. npm install -g node-gyp

  3. npm install -g nan

  4. npm install
    (inside package/project directory)

  5. result: failed (same as mentioned before)


#6

I managed to install and build a grammar from scratch on Windows, and I ran into a couple of issues along the way.

Before starting, I installed Python2.7; it crashed a couple of times, but that may have been my fault (turns out laptops don’t run very well on 0% battery. Who knew?). I also installed nvm (for Windows) and node version 8.9.3 - I’ll get back to this in a guide.

Moving on. First, I had to install build tools like you say you did in your edit. I had to do this in an administrator prompt. Interestingly, it looks like it would install python2.7 if it wasn’t already there. I don’t know if other programs would be able to find it, but it’s good to note.

Next, I tried running npm install on the directory containing just the package.json file I pasted above. It threw and error about missing some file… The following permissions change worked to solve it (I was running npm install as a regular user; it may have worked if I was administrator for that?). Anyway, the change was difficult to find. I ended up ticking things until it worked. Maybe a proper Windows person knows more.

I then ran npm rebuild again, and so far as I remember it worked (there were a lot of warnings, but no errors).

Up next was writing the grammar. So I copied over this file into the project directory.

Then ran .\node_modules\.bin\tree-sitter generate; no error. (note: I don’t know how to add relative paths to Windows PATH, so that’s why I used the path to the binary to run that command).

At node-gyp configure I got a “node-gyp not found error” So I ran npm install -g node-gyp and it installed without issue.

I then ran node-gyp configure again, and everything was OK.

Almost finished now; just needed to grab another test file from the tree-sitter-biber repo. I grabbed this one and placed it in the corpus folder (need to create it) of the dummy grammar I’m walking through here.

Finally, I ran .\node_modules\.bin\tree-sitter test, and it printed the results of checking all the rules.


#7

And now I’m getting off Windows as quickly as possible.


#8

I never ran into a nan related error either. So I really don’t know what went wrong with yours. Maybe try deleting the node_modules folder and trying again from the first few steps?

You don’t actually need to remove Python 3; just make sure that the python command points to Python 2. I didn’t test this on Windows, but I believe the python, python3, and python2 commands all exist?

Also, and I will explain better in a guide, I use nvm to install node because it lets you install multiple versions. The latest version is 10.x.x, but Atom (well, Electron) uses 8.9.3. When I first started, I ran into compatibility errors when trying to load the package into Atom, hence the trying to stick to the right version here. If you don’t need nodejs for anything else, installing it directly should be fine.


#9

I’ve started writing the gist here. The requirements and setup are pretty much done, but I may need to add more if the nan error persists for @danPadric.

Now I’ve got to decide what grammar to write. I’m thinking I’ll stick with biber, because I sort of know how to write that one. And hey; it’ll help me fix it up too.


#10

speechless
recovering…

Wow! you have put much effort in analyzing my problem
and then make a well written document.
More! in the middle of your night to return here and report on it!!
That is far beyond the call of duty - then as a volunteer?!
speechless
recovering…

I just had to log in and say -
Thank you very much.




It is a mix batch of news from my side.

The bad news is that my Windows 10 machine refuses
to give me an error free result. I have spent time on this looking at
possible causes and remedies. My google-foo seems to be messed up - a lot
of nothingness.

I plan on going through the steps again one final time

  • using your notes ‘religiously’ -
    to get npm install to give positive results on my project.

There is a Windows 7 machine that I have access to.
I tried the npm install on this machine and it went smoothly!
This PC has the tools already installed for a previous Atom-related issue.
There was no need to make additional node package installations.
Additionally, the JSON file I tried to use on Windows10 now worked.

I am not sure what is different in tool-sets
between the Win7 and Win10 machines to cause the change.

The steps done in Win7:

  1. npm install --global --production windows-build-tools
    (was done on a previous occasion with elevated command line)
    [SOLVED] " Cannot find module '..\build\Release\pty.node' "

  2. Place JSON file in path D:\zoo-tree.
    Change directory in command line to this project path.

  3. npm install
    (inside package/project directory)




As a side note -
Please tell people in your notes that nvm is Node Version Manager.
:blush: The last of my google-foo was used up on this shorthand.
Perhaps best if I do not announce other proofing issues…
I run the risk of sounding critical, unkind and ungrateful.

I hope to let you know of anymore progress if my plans pen out.
Best regards.


#11

Hello.

I remain stuck in the mud. I have followed your instructions & using the JSON sample. Yet cannot move beyond the error when doing npn install.

I have started it for a 2nd time (follow your instructions) and this time wrote notes to document the accumulated ‘knowledge’.

Preliminary notes

Progress report

An attempt was made to create a Tree-Sitter project, following the instructions in Guide to your first Tree-sitter grammar. The target OS is Windows 10 Prof. V1803 Build 17134.167.

Requirements

  1. All command line operations were executed within an elevated command line. All command executions mentioned is done in the command line (command prompt).

  2. Installation of Node Version Management (nvm) as per #. nvm was used to install Node V8.9.3. The command line was closed and re-opened to give the newly set environment variables a chance to load.

  3. To activate nvm, the command nvm on can be executed. Thereafter node --version confirms that the node command is available on the PATH.

    In this case the node --version did not execute. Traces of a previous Node application was still on the system. These instructions were followed.

  4. Installation of build tools using the install --global --production windows-build-tools.

    For this case, Python V2.7.15 was already installed and the python command is available from the path. The build tools did not re-install Python. A copy was made of python.exe and renamed to python27.exe. This file is associated with python commands: npm config set python "C:\Python27\python27.exe"

  5. Installation of node-gyp -> npm install -g node-gyp --save.

    Installation completed as node-gyp@3.7.0 and confirmed with node-gyp --version. Followed by node-gyp install while in the project path.

------ not complete --------

Executing npn install in the project path gives the error as stated before. Looking at the instruction given here, seems to suggest that a binding.gyp file is required. Adding this into the project or in the node-gyp install path makes no difference.

I noticed the the error notes node-gyp as version 3.6.2 while the global install of node-gyp is V3.7.0. See the first posting.




Linking a copy of the biber grammar into Atom as a package does not work for me either. Used: apm link "D:\github\tree-sitter-biber". Atom gives an error.


My time has run out for spending time on this case. My day-job is deploying me onto hostile task that needs my full attention to prepare. Please let us put a pin into this issue.


#12

OK, I’ve updated the guide to have a “quick check” section. Assuming you’ve got the build tools and node-gyp already installed (and it sounds like you have if it was complaining about a missing binding.gyp), try these steps (when you get an opportunity again)

  • Clone this grammar repo
    git clone https://github.com/Aerijo/tree-sitter-quick_check
    
  • Navigate to the root of this project in a terminal / command prompt
  • Run npm install
  • Add ./node_modules/.bin to your PATH
    • If you don’t add this, prefix all tree-sitter commands with ./node_modules/.bin/; e.g., tree-sitter generate becomes ./node_modules/.bin/tree-sitter generate
    • Windows users need to replace forward slashes with back slashes.
  • Run tree-sitter generate (among other things, this makes the binding.gyp file for the next step)
  • Run node-gyp configure (required before you can run build)
  • Run node-gyp build
  • Run tree-sitter test

If it still doesn’t work, then I really give up.


#13

Thank you. I have seen the updates -> Nice work.

As for the your suggestions - I will not be able to test those for a number of days.
It is my intent to let you know as soon as I have tried it.