Big Data for offline app


#1

Hello Everybody!
i got a question about handling Big Data in offline application in Electron.

i made my application which handles a bunch of data (actually a 150MB Sqlite Database)
i’m handling the sqlite data with sql.js library, because the sqlite npm library doesent’t pass the buil phase (even with some tricks found online)

what i’d like to know is if this is the best solution to handle offline data, with update capability.

my app actually connects to my server, syncs with the online database, and allow the user to work offline.
the sync is made by downloading the whole DB every time, because if i perform a massive INSERT statement (even with TRANSACTION, pragma and everything else) the sql.js library hase some issues with memory.

now they’re asking me to add more data, so the DB size will grow.

is any other option that may i evaluate?!

i was evaluating everything, from localstorage to PouchDB, but i don’t want to bring down everything without good reasons…

any help will be appreciated
:heart_eyes:


#2

no one has some ideas?!


#3

To begin with, this is not necessarily an electron question as you can use in electron basically anything that runs in nodejs or the browser context. You may find more useful information regarding the choice of a database scheme or specific db implementation (such as sql, couchdb, pouchdb, localstorage) in database related forums.

You mentioned pouchdb, which is a great solution if you want to use CouchDB as the backend, because it is maintained actively and offers as great API to work with. I can certainly recommend it, if you want to sync data when online and have the ability to use an offline DB inside electron when there is no DB connection.

But in the end it depends what you want to do with the DB, what your existing infrastructure looks like, whether you want to keep it that way or if the project is in an early enough stage to change the DB and if thats the case what DB you exactly need in your situation.

I am afraid you won’t find an easy solution here, because of all the variables involved choosing the best DB for your application.


#4

Thanks for the answer!

actually everything is held (online) by a SQLServer DB. I didn’t choose Pouch/Couch for the differences that will come and how the data is structured (i need a lot of filters, joins etc.)

i was wondering that some other DB structure could be more more “efficent” for big data, speed and consistency, or someone has a solution for the npm sqlite library, or some wizard had a magic wand :smiley: