Concurrent read access by R processes

[riccardopinosio]

Hello,

At the end of the monetDB lite page I read:

"MonetDBLite does not allow multiple R sessions to connect to a single database concurrently. As soon as a single R session loads an embedded server, that server is locked down and inaccessible to other R consoles."

I was wondering what is the technical reason for this limitation and whether this limitation could be removed by, e.g., having the accessed tables be read-only. The reason I am interested is that monetDB with concurrent read access by R processes would be a killer "frontend db" for R powered web apps using, e.g., openCPU, which allows for api calls to an R package which are served by R processes (it would be useful for shiny too, of course). Any thoughts?

Riccardo.

[hannes]

Hi Riccardo,

I think an all-read-only mode could be done. Specifically, allowing multiple clients to read the same data, but only if all are in read-only mode. Would this work for you? The technical reason is that much of MonetDB's state is in memory, and there is no means of broadcasting this state between processes. For example, if someone creates a table in one process, this fact would need to be communicated to the other processes, for example by sending messages. If all of the processes are in read-only mode, this can't happen and the only issue are temporary structures (intermediate results in queries) that would have to be stored somewhere. But MonetDB has support for putting those in a separate folder, so it should be possible.

Hope this helps,

Hannes

[dselivanov]

That would be awesome addition!

[riccardopinosio]

Indeed, I also think it would be awesome to have that. I think the use case for such a feature is pretty clear: you have a shiny app/open CPU app that needs to read data to display to users, and you don't want to install a full fledged monetdb (I assume you can have concurrent access with the full fledged monetdb?). At the moment I solved the problem by installing redis DB which is pretty fast but monetdb would be a better storage solution if this was possible.

[caewok]

Yes, a read-only mode to allow multiple client access would be a great addition. This might address the issue of allowing multiple R threads to access the database, which currently cannot be done in MonetDBLite. For example, when running multiple models of a single dataset in multiple threads, it would be more memory efficient to load the dataset as needed in each thread, instead of loading it at the beginning and passing it to each thread (which tends to result in multiple copies in memory).

[gminorcoles]

Plus one

[mbedward]

Another vote for this enhancement

[cboettig]

An all read-only mode would be great