disable COPY_STACKS

[vtjnash]

While COPY_STACKS is nice for keeping memory allocation somewhat limited (due to inference and codegen requirements), it is bad for c integration. Since we have such a nice cfunction, it is a shame that COPY_STACKS significantly complicates/limits what you can do inside of those functions (e.g. no calling the same library on a different task inside a cfunction). Other benefits of disabling it include avoiding making backup copies of the stack for each task, and copying that memory with each task switch.

therefore, my proposal is to
(a) disable copy-stacks
(b) reintroduce the code for running inference / codegen in a separate thread
(c) only do the thread context-switch when not running --build mode

[JeffBezanson]

I wouldn't be so sure. There are two big problems: (1) how are you going to grow stacks? (2) some C libraries get confused if the stack pointer doesn't point to the actual process stack. I don't yet fully understand the problem with Gtk, but it might be possible to address it by leaving more frames in place instead of overwriting basically the whole stack on each switch.

What does --build mode have to do with it?

[vtjnash]

dump.c can't serialize a Task

Gtk wants to store a small amount of data on the stack, with the assumption that that data will be there in future calls from the same thread that are lower on the call stack (it is tracking recursion)

[JeffBezanson]

Ok, I think we can handle that by adjusting down the stack base used for copying. JL_SET_STACK_BASE is the current way to do this, but there might be something better we can do.

[vtjnash]

also, by "some libraries" did you mean pthreads, which asks that you tell it if the stack moves http://pubs.opengroup.org/onlinepubs/009695399/functions/pthread_attr_setstack.html (i think this is necessary for it to guess which thread is running, for TLS lookup)?

On Windows, we could just switch to using Fibers (http://msdn.microsoft.com/en-us/library/windows/desktop/ms682402(v=vs.85).aspx). On linux, I have seen other client libraries that claim to do this, so it is certainly possible.

What does JL_SET_STACK_BASE do? I think Gtk does something like this (where chain is in TLS):

void *chain;
void c_function(void *data) {
    char* link[2];
    link[0] = chain;
    link[1] = data;
    chain = link;
    julia_callback(); /* might call c_function */
    chain = link[0];
}

[JeffBezanson]

JL_SET_STACK_BASE maintains a jmp_buf to use as the top of the stack. To switch stacks, we unwind to there and copy in the new task's stack.

Most C code that does stack switching knows a decent static bound on stack sizes. But we'd like to be able to run arbitrary code in any task, which requires stacks to grow. The other problems are much less serious than this one.

[vtjnash]

that's what I thought, but that only reduces the need to store the very top of the stack, but doesn't (can't) address the problem of c code storing data further down the stack

right, but the ability to run arbitrary code also means that the stack needs stable memory addresses, since C libraries may try to store information there (see the code above for an example).

committing memory (but not touching it) is essentially free, especially in a 64-bit system. it'll inflate julia's reported memory allocation, but not actually impact the usage. we could consider setting up fiber pools so that when one task dies, a new task can reuse it's stack area, saving a bit of work for mmap to zero the reused pages.

[JeffBezanson]

Ok, I have no problem with making sure !COPY_STACKS still works, giving each task ~4-8MB by default and pooling them. This could become the default on 64-bit linux at least (I'm just less sure how other systems will behave). Using fibers on windows could be really good too.

Another problem I've seen is systems with security features that prevent you from arbitrarily switching the stack pointer. Some distros may still need to build with COPY_STACKS.

I still don't understand your step (c). This change doesn't make it much easier to serialize stacks. We could simply disallow task switches during --build (throw an error?), but that's too strict since you might do task switches but ultimately not try to serialize any suspended tasks.

[vtjnash]

The concern with step (c) is that you can't run inference / code-gen in a separate task during build, since dump.c will throw an error when you try to serialize it

[JeffBezanson]

If necessary, that could be handled with a special built-in task that we know not to serialize.

[vtjnash]

Another problem I've seen is systems with security features that prevent you from arbitrarily switching the stack pointer. Some distros may still need to build with COPY_STACKS.

the stack-smashing protector? or something else? I don't think the SSP is a concern here

[JeffBezanson]

It was something else I believe.

[vtjnash]

Hardware Stack Protection (NX bit)? That only matters if you try to execute the stack (and it's orthogonal to COPY_STACK)

[KristofferC]

Is this still relevant?

[yuyichao]

YEs

[Keno]

Addressed by the partr task work?