Skip to content
This repository has been archived by the owner on Apr 22, 2023. It is now read-only.

Writing to files in loops seems to be capped at 250 file.write calls. #1

Closed
n8agrin opened this issue Jun 3, 2009 · 1 comment
Closed

Comments

@n8agrin
Copy link

n8agrin commented Jun 3, 2009

I wrote a simple loop last night to test File I/O and noticed the following behavior. If I created a loop like so:

var times = [];
for (var i=0,j=251; i<j; i++) {
    var t1 = Date.now();
    var file = new node.fs.File();
    file.open('test.log', 'a+');
    file.write(i + ' ' + new Date() + ' Hello World!\n');
    file.close();
    times.push((Date.now() - t1));
}

'file' would contain only 250 new lines of text, while the array 'times' would show it's length to be 1000. The number of new lines added to the file seems fixed at 250. Setting the loop to 249 produced a times array with 249 items and a file with 249 new lines. Setting the loop to 251 produced a times array with 251 items and a file with 250 new lines.

My C++ is very poor, but my guess is that each event has a queue which is hard coded at 250 and my loop over flowed the queue.

Expected:
In a loop you should be able to write to files as much as you like. A web-server that is getting hit many times per second might overflow a 250 limit.

@ry
Copy link

ry commented Jun 4, 2009

So, remember: this is not sequential programming. In this example what you're doing is immediately sending the operating system 250 "open()" system calls to the same file. Then you're writing to them in parallel. This is rather dangerous because it isn't guaranteed that each line is going to be printed in the right order - it could even be that characters from the various lines get mixed up with each other.

I think you probably hitting some sort of maximum number of times a single file can be open at once limit on your operating system. On Linux for example, the program handles a thousand concurrent writers.

cole-gillespie pushed a commit to cole-gillespie/node that referenced this issue May 24, 2011
bash-3.2# node-waf build
Waf: Entering directory /calipso/node_modules/node-expat/build'
[1/2] cxx: node-expat.cc -> build/default/node-expat_1.o
In file included from /usr/local/include/node/uv.h:40,
from /usr/local/include/node/node.h:25,
from ../node-expat.cc:1:
/usr/local/include/node/uv-unix.h:27:19: error: ev/ev.h: No such file or directory
In file included from /usr/local/include/node/node.h:25,
from ../node-expat.cc:1:
/usr/local/include/node/uv.h:145: error: ‘ev_timer’ does not name a type
/usr/local/include/node/uv.h:158: error: ‘ev_idle’ does not name a type
/usr/local/include/node/uv.h:158: error: ‘ev_io’ does not name a type
/usr/local/include/node/uv.h:158: error: ‘ev_io’ does not name a type
/usr/local/include/node/uv.h:158: error: ‘ev_prepare’ does not name a type
/usr/local/include/node/uv.h:158: error: ‘ev_check’ does not name a type
/usr/local/include/node/uv.h:158: error: ‘ev_idle’ does not name a type
/usr/local/include/node/uv.h:158: error: ‘ev_async’ does not name a type
/usr/local/include/node/uv.h:158: error: ‘ev_timer’ does not name a type
Waf: Leaving directory/calipso/node_modules/node-expat/build'
Build failed: -> task failed (err nodejs#1): 
{task: cxx node-expat.cc -> node-expat_1.o}
rpetrich pushed a commit to rpetrich/node-ancient that referenced this issue Dec 15, 2011
choonkeat pushed a commit to choonkeat/node that referenced this issue Mar 28, 2012
Bug nodejs#1 occurred when trying to use process.mixin on a function and
produced a fatal exception.

Bug nodejs#2 occurred when trying to do a deep merge with an object containing
one or more objects with a nodeType property. In those cases the deep
copy for this part of the object was not performed and a shallow one was
performed instead.

Both of these bugs were artifacts of the jQuery.extend port.
This issue was closed.
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants