Find files programmatically and fast?


#1

I’m looking for a way to programmatically find files under my projects. This is for selecting which project definition file to boot up in an IDE-like external process: https://github.com/ensime/ensime-atom

I couldn’t find any API in Atom so I resolved to do it with node’s fs but even there I couldn’t find any recursive search so I ended up with a third party lib. The code became kindof complex but worse, it became really slow. I basically just need to find ALL files named “.ensime” under any of my projects root folders. This is what I have:

selectAndBootAnEnsime: ->
    #recursive-readdir is callback based, have to promisify to flatten
    recread = require 'recursive-readdir'
    Promise = require 'bluebird'

    recreadPromisified = Promise.promisify(recread)
    dirs = atom.project.getPaths()

    promises = dirs.map (dir) ->
      recreadPromisified(dir, [dotEnsimesFilter])

    promise = Promise.all(promises)
    promise.then (dotEnsimesUnflattened) =>
      dotEnsimes = _.flatten(dotEnsimesUnflattened)
      new SelectFile(dotEnsimes, (selectedDotEnsime) =>
        @startInstance(selectedDotEnsime)
      )

I guess there’s cleanup to do, but fact remains that it’s pretty slow for some of my projects which have looots of files under them. Basically, I’m asking - isn’t there any index of files reachable from an API of Atom? I’m looking at fuzzy-finder: https://github.com/atom/fuzzy-finder/blob/master/lib/path-loader.coffee
but can’t seem to understand it…
Any tips?


#2

The documentation for the Task class pointed me ultimately to the scandal module. Perhaps it could be helpful? It looks like the stuff in Atom that does what you’re asking about ultimately use scandal.


#3

@leedohm Thanks! Awesome, looks to be just what I needed!

At least to fix performance. I guess I still need to workaround the event/callback thing by promisifying those events… Don’t think there’s a built in way as for callbacks… I’ll be back with findings!


#4

Everything is just a google away :smile:
https://www.npmjs.com/package/bluebird-events


#5

Weird. Maybe it’s just something bottlenecky with the way I made promises and flattened, don’t know much about this in js-land, but it still takes like 5 seconds for me with this:

{PathScanner} = require 'scandal'
    dirs = atom.project.getPaths()
    promises = dirs.map (dir) ->
      scanner = new PathScanner(dir,
        inclusions: ['**/.ensime']
        includeHidden: true
        exclusions: ['.ensime_cache/', '.git', 'target']
        )
      findings = []
      scanner.on 'path-found', (path) ->
        findings.push path
      new Promise (resolve, reject) ->
        scanner.on 'finished-scanning', ->
          resolve findings
        scanner.scan()

    Bluebird = require 'bluebird'
    promise = Bluebird.all(promises)

    promise.then (dotEnsimesUnflattened) =>
      dotEnsimes = _.flatten(dotEnsimesUnflattened)
      new SelectFile(dotEnsimes, (selectedDotEnsime) =>
        console.log(['selectedDotEnsime: ', selectedDotEnsime])
        @startInstance(selectedDotEnsime)
      )

I’m gonna have to take a new look at it tomorrow.