Nested anonymous functions not propagating to workers

[ivirshup]

Hi,

I think I've found a bug? Nested anonymous function aren't propagating to workers with @everywhere:

julia> addprocs(1)
1-element Array{Int64,1}:
 2

julia> @everywhere f = (x->(y->y+1)(x) + 1)

julia> remotecall_fetch(f, 2, 1)
ERROR: On worker 2:
UndefVarError: 6#8 not defined
 in #5 at ./none:1
 in run_work_thunk at ./multi.jl:722
 [inlined code] from ./multi.jl:1018
 in #253 at ./task.jl:59
 in remotecall_fetch(::Any, ::Base.Worker, ::Int64, ::Vararg{Int64}) at ./multi.jl:808
 in remotecall_fetch(::Any, ::Int64, ::Int64, ::Vararg{Int64}) at ./multi.jl:811
 in eval(::Module, ::Any) at ./boot.jl:267

julia> versioninfo()
Julia Version 0.5.0-dev+3050
Commit c96f322 (2016-03-08 04:08 UTC)
Platform Info:
  System: Darwin (x86_64-apple-darwin15.3.0)
  CPU: Intel(R) Core(TM) i7-4870HQ CPU @ 2.50GHz
  WORD_SIZE: 64
  BLAS: libopenblas (USE64BITINT DYNAMIC_ARCH NO_AFFINITY Haswell)
  LAPACK: libopenblas64_
  LIBM: libopenlibm
  LLVM: libLLVM-3.7.1

Update:

I've found a more general example:

julia> addprocs(1)
1-element Array{Int64,1}:
 2

julia> remotecall_fetch((x->(y->y)(x)), 2, 1)
ERROR: On worker 2:
UndefVarError: 2#4 not defined
 in #1 at ./none:1
 in run_work_thunk at ./multi.jl:744
 [inlined code] from ./multi.jl:1040
 in #255 at ./task.jl:59
 in remotecall_fetch(::Any, ::Base.Worker, ::Int64, ::Vararg{Int64}) at ./multi.jl:830
 in remotecall_fetch(::Any, ::Int64, ::Int64, ::Vararg{Int64}) at ./multi.jl:833
 in eval(::Module, ::Any) at ./boot.jl:267

[amitmurthy]

The following work:

julia> @everywhere println(f(1))
3
    From worker 2:  3
    From worker 3:  3

julia> remotecall_fetch(x->f(1), 2, 1)
3

It appears that f is indeed defined on all workers. In the case of remotecall_fetch(f, 2, 1) while one would expect only the symbol and module to be serialized, it is actually serializing the locally defined function.

cc: @JeffBezanson

[ivirshup]

Came across another (probably more common) case:

pmap([rand(2) for i in 1:2]) do x
    map(y->typeof(y), x)
end

This generates a pretty long error report so I've put an example of running it here.

edit: corrected code.

[JeffBezanson]

The issue here is that nested functions now generate multiple top-level functions. So the immediate function you're trying to send refers to globals in Main that are not being sent.

I think the right solution is to look through the IR for the function when it's serialized, and send over all its dependencies. This could also potentially let us remove the localize_vars hack. We can also keep track of which items from Main we've sent to which processors and avoid re-sending, which would be a helpful optimization.