Directory structure questions about modules

I'm new to both Electron and Linux and so I believe my issues are a result of my ignorance of both. Coming from a Windows background I never realized how many things my OS was managing for me that Linux forces me to learn how to do manually.

I'm not sure but I don't think I'm managing my file system correctly. I have allowed npm and apt to install stuff in it's default locations and I believe that may have been a mistake. For starters, there is a folder in my home directory called node_modules and 2 files, one called package.json and package-lock.json that I think npm created when I installed a couple of electron modules, however; if I look in "usr/lib/nodejs/npm" there is a node_modules directory in there as well. I think I may have mistook how npm install is supposed to be used. Without thinking about it I deleted the package.json file in my home directory and now I get a series of warnings about a missing package.json file whenever a install a node module with npm.

    Is the npm install command only supposed to be used when I'm working in the directory of an electron app I'm working on?
    
    Should I be installing node modules in the "usr/lib/nodejs/npm/node_modules" directory?
    
    Do I need to recreate the package.json file in my home directory?
    
    Have I just made a big mess of everything?
    
    Thanks,
    Mark

Since you are in Linux take advantage of its command line interface (cli).

You seem to be unaware that npm can install packages either globally or locally (e.g. in your Electron app).

If you look in your $HOME/.atom/packages directory (for example - ensuring that you view hidden files) every package contains its own local package.json and its own local node_modules folder. Same applies to Electron apps.

If you deleted your $HOME/package.json file in error then this disables your global installation of node_modules installed in $HOME … however if you are very lucky you might find still the deleted package.json in your Rubbish Bin to restore it. If not then you will need to reinstall node in global install mode. You will also need to reinstall any added modules in global since this updates core package.json for every such installation (see “dependencies”).

In your Linux shell terminal run the command node --help and npm --help to read the options and also refer to node.js documents to learn about global usage and local usage.

If you run the command sudo locate package.json you will see countless such files scattered throughout your filesystem

Sorry that my original post was just a couple of really long strings. I wrote it in a text app before I posted it because I wasn’t sure how long my window would remain valid while open online.

Anyway, I get that by your reply that it is not unusual to find node_module directories in more than one location on my computer. I also get that npm is indeed installing them on my computer in order for me to use them at any time in the future from my local install and that I do not need to install it each module individually from the web in what ever electron application I may create.

Unfortunately, I’m bummed that I need to reinstall all the modules that are in my HOME directory because like a meathead I threw away my package.json file. I thought that the package.json file was only for electron apps and I didn’t see why one belonged sitting there in the root of my HOME directory since it was not an electron app. I figured I had done something wrong or mistook the way that npm was supposed to work and so I threw it away. Then I started getting those warnings about not having a package.json file when I installed a new node module and I knew that I really didn’t have a grasp on what I was doing then.

So I have to reinstall node as well? I did originally install node in global mode I believe because at one point I had to go back and reestablish group and user ownership of something because apt or whatever I was using was complaining and refusing to do certain things because of root ownership set on stuff.

Anyway, I sure am getting a workout in the terminal with Linux. It’s becoming quite an education.

Thanks,
Mark

If you did not find the deleted file in Rubbish Bin then yes you need to reinstall (although there are ways of recovering recently deleted files).

Another useful command to run in terminal is simply history.
This will give you a date ordered list of commands you have run and you should be able to find the node modules you installed to go through it again.

To narrow history search you can run the command history | grep npm to only print those npm commands.

And as you start to learn more about the command line you might run history | grep npm > myhistory.txt which you could use to build a bash script to re-install multiple modules.

The $HOME/node_modules folder can also be inspected to see which modules were installed (look at date order). In Ubuntu I use Krusader as my file manager which makes these tasks easy (although Krusader does come with heavy KDE baggage)

[P.S.] Do not delete $HOME/node_modules but you might disable by temporarily changing its name to $HOME/__node_modules. Then when you reinstall you can use this as a reference when re-installing modules. When your fresh installation is installed you can then delete this old folder. Another tip to learn in Linux is how to frequently backup.

This isn’t a Linux-specific issue, first off. How you use Node itself is a little different between the platforms, but installing modules is basically the same.

Yes, it is, unless you use the --global/-g switch. That switch will install a module globally, which means that Node can access it no matter which project you’re working in (it has the same level of availability as the modules built into Node, like fs and path). You might want to install electron globally, for example, if you have multiple Electron projects and don’t want multiple copies of it on your hard drive (it’s fairly big).

Should I be installing node modules in the “usr/lib/nodejs/npm/node_modules” directory?

No. That folder is for NPM to use. Until or unless you become a Linux power user, you should avoid modifying anything outside of /home/ or /etc/ (sometimes you might be following a tutorial that tells you to edit one of the text files in there, like /etc/hosts, and those you can follow without worrying about messing something up).

Side-note about Linuxiness: the leading / is actually really important. You won’t confuse anyone in the know by leaving it out of a path like /usr/lib/, since that’s a prominent system folder, but the / indicates the root of the operating system’s file tree, so it clearly tells people where you’re starting from.

I have a folder for all of my projects just called ~/github/, so that I always know where my repositories are (Tux Tip™: ~ is shorthand for your user folder, and it works in the command line for both Linux and Mac). Once you decide where you want to put it and make a folder for your project only, you should enter that folder in the terminal and type npm init. NPM will guide you through setting up a basic package.json, and then you’ll be able to install modules. You should use the --save/-s switch to tell Node to add each module you install to the dependency list stored in package.json (for example: npm i -s pad-left).

Well, I’ve uninstalled and reinstalled node and npm. I used the instructions from nodesource to get their package and install it. I’ve noticed that their package does not create any directories. It only installs binaries in “/bin”. I installed electron-log using npm and it created a new “node_modules” directory and package-lock.json file in my $HOME directory, but it did NOT create a new package.json file. Should I just do “npm init” in my $HOME directory and create one and call the app $HOME?

Also, I was wondering if anyone who has a “node_modules” directory in their $HOME directory would check their package.json file in that same directory and tell me if there is any listing in there that identifies it as belonging to the $HOME directory? I figure I can just place a new package.json file in my $HOME and solve my problem that way.

I searched around for a possible solution to possibly reverse build package.json from $HOME/node_modules directory (I hope that you retained a copy of your original node_modules folder as advised earlier).

I found this but I have not tested it on my own global installation.

To answer your question there is no reference to $HOME in my $HOME/package.json.

[P.S.] More tips found here. e.g. shrinkwrap

I sure appreciate all the help I’m getting here. It’s crazy, I thought it might be less complex building platform independent desktop apps using web technologies. Oh well.

Anyway, yes, I did indeed follow your advice d_l and renamed my node_modules directory and even saved the package-lock.json file inside of it. When I reinstalled node and npm it created a new node_modules directory and a new package-lock.json file in my $HOME. It just didn’t create a new package.json file. I wonder why? Anyway, thanks for the links

Globally installed packages are handled differently for each OS, since they have to be user-specific and available through the shell from anywhere. For just working with individual repositories that don’t use native code, it’s not so complex.

That sounds o.k. You should have package.json at $HOME/package.json but not as you seem to expect inside node-modules. Within each subfolder (module) inside node_modules folder you will expect to find other package.json files for each module.

In $HOME run the command npm list to see the hierarchy of modules and dependencies.

Musing here. Since I do not have any package.json file immediately inside $HOME/node_modules (as I explained earlier) I wonder if the file you refer to is in fact the missing package.json which you wrote you deleted?

Oh no, I didn’t have a package.json file inside of the node_modules directory, it was actually outside of it in my $HOME directory… and I deleted it. Now, whenever I install a module that isn’t specifically inside of and part of an application I get ERR Warnings from npm about this missing package.json file. I guess I’m just going to create a new package.json file in my $HOME directory and see what happens.

I was concerned that I might mess something else up by just plopping a new package.json file in there since I don’t really know what I’m doing yet.

Well I just did a npm init in my $HOME directory and that seems to satisfy the npm install issue.

On another note: I took the bait at the bottom of your first post DamnedScholar and did a google search on "pad-left " because I was wondering what on earth a whole module devoted to something being called pad-left would be all about. Ouch. Now that really makes me wonder what I’m getting myself in to. But thanks for the heads up anyway.

1 Like

There are a lot of smart people with much more invested than you or I who are working to ensure that the pad-left incident doesn’t occur again. It remains an educational story about the dangers of distributed dependency ecosystems and a computer science parable to make people think about the positives and negatives of the choices they make. And as an example, it’s ubiquitous and generic enough to make a good sample command, while having history that people who have been working with NPM for a while would recognize. :slight_smile:

The best way to protect yourself against another pad-left uses a strategy that turns out to be a pretty good practice overall for projects larger than a single script or loop: having separate dev and production environments. These can be as simple as two folders on the same computer, or as elaborate as containers running on cloud servers; one contains a completely functional version of your program, while the other one is what you change constantly. When you update anything, including dependencies, you test it in your dev environment first. If a pad-left fiasco were to happen again, anyone with a production environment set up will have access to the working version of the code from the offending package.