Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Serialization of nested anonymous functions. #19456

Closed
amitmurthy opened this issue Nov 30, 2016 · 14 comments
Closed

Serialization of nested anonymous functions. #19456

amitmurthy opened this issue Nov 30, 2016 · 14 comments
Labels
bug Indicates an unexpected problem or unintended behavior parallelism Parallel or distributed computation regression Regression in behavior compared to a previous version

Comments

@amitmurthy
Copy link
Contributor 

julia> c = x->x
(anonymous function)

julia> d = x->c(x)
(anonymous function)

On 0.4:

julia> pmap(d, 1:2)
2-element Array{Any,1}:
 1
 2

On 0.5:

julia> pmap(d, 1:2)
ERROR: On worker 2:
UndefVarError: c not defined
 in #3 at ./REPL[2]:1

Note that pmap(c, 1:2) works.

Making them local results in a different error:

julia> function wrapper()
           c = x->x
           d = x->c()
           pmap(d, 1:2)
       end
wrapper (generic function with 1 method)

julia> wrapper()
ERROR: On worker 2:
MethodError: no method matching (::Base.Serializer.__deserialized_types__.##5#7)()
@amitmurthy amitmurthy mentioned this issue Nov 30, 2016
Open
@amitmurthy
Copy link
Contributor Author

cc: @JeffBezanson , @vtjnash

@amitmurthy amitmurthy added bug Indicates an unexpected problem or unintended behavior parallelism Parallel or distributed computation labels Nov 30, 2016
@amitmurthy amitmurthy mentioned this issue Dec 1, 2016
Closed
@amitmurthy amitmurthy added the regression Regression in behavior compared to a previous version label Dec 3, 2016
@bdeonovic
Copy link
Contributor

Any idea if this can be fixed?

@amitmurthy
Copy link
Contributor Author

I am pretty sure it can be. Waiting on Jeff or Jameson as they are most familiar with the serialization codebase - unless someone else takes a shot it first.

@bdeonovic
Copy link
Contributor

Still no ETA @JeffBezanson , @vtjnash ?

@amitmurthy
Copy link
Contributor Author

Soon. #19594 will fix this.

@amitmurthy
Copy link
Contributor Author

Hmmm, spoke a bit too soon. #19594 fixes the issue when the closures are global. It does not address the second case, i.e. local closures.

@amitmurthy
Copy link
Contributor Author

My bad. The wrapper example above has an error. It should be

function wrapper()
           c = x->x
           d = x->c(x)
           pmap(d, 1:2)
       end

In fact it works on 0.5 too.

@bdeonovic
Copy link
Contributor

bdeonovic commented Jan 13, 2017 •
edited
Loading

How about:

julia> function wrapper(y)
                  c = x->x
                  d = x->c(x)
                  d(y)
       end

julia> pmap(wrapper,1:2)
ERROR: On worker 2:
UndefVarError: #wrapper not defined
 in deserialize_datatype at ./serialize.jl:823
 in handle_deserialize at ./serialize.jl:571
 in deserialize_msg at ./multi.jl:120
 in message_handler_loop at ./multi.jl:1317
 in process_tcp_streams at ./multi.jl:1276
 in #618 at ./event.jl:68
 in #remotecall_fetch#606(::Array{Any,1}, ::Function, ::Function, ::Base.Worker, ::Int64, ::Vararg{Int64,N}) at ./multi.jl:1070
 in remotecall_fetch(::Function, ::Base.Worker, ::Int64, ::Vararg{Int64,N}) at ./multi.jl:1062
 in #remotecall_fetch#609(::Array{Any,1}, ::Function, ::Function, ::Int64, ::Int64, ::Vararg{Int64,N}) at ./multi.jl:1080
 in remotecall_fetch(::Function, ::Int64, ::Int64, ::Vararg{Int64,N}) at ./multi.jl:1080
 in #remotecall_pool#689(::Array{Any,1}, ::Function, ::Function, ::Function, ::WorkerPool, ::Int64, ::Vararg{Int64,N}) at ./workerpool.jl:93
 in remotecall_pool(::Function, ::Function, ::WorkerPool, ::Int64, ::Vararg{Int64,N}) at ./workerpool.jl:91
 in #remotecall_fetch#692(::Array{Any,1}, ::Function, ::Function, ::WorkerPool, ::Int64, ::Vararg{Int64,N}) at ./workerpool.jl:124
 in remotecall_fetch(::Function, ::WorkerPool, ::Int64, ::Vararg{Int64,N}) at ./workerpool.jl:124
 in (::Base.###697#698#700{WorkerPool,#wrapper})(::Array{Any,1}, ::Function, ::Int64, ::Vararg{Int64,N}) at ./workerpool.jl:151
 in (::Base.##697#699)(::Int64, ::Vararg{Int64,N}) at ./workerpool.jl:151
 in macro expansion at ./asyncmap.jl:63 [inlined]
 in (::Base.##755#757{Base.AsyncCollector,Base.AsyncCollectorState})() at ./task.jl:360

...and 1 other exceptions.

 in sync_end() at ./task.jl:311
 in done(::Base.AsyncCollector, ::Base.AsyncCollectorState) at ./asyncmap.jl:124
 in pump_source(::Base.AsyncGenerator, ::Base.AsyncGeneratorState) at ./asyncmap.jl:185
 in next(::Base.AsyncGenerator, ::Base.AsyncGeneratorState) at ./asyncmap.jl:201
 in _collect(::UnitRange{Int64}, ::Base.AsyncGenerator, ::Base.HasEltype, ::Base.SizeUnknown) at ./array.jl:282
 in #pmap#714(::Bool, ::Int64, ::Void, ::Int64, ::Float64, ::Function, ::Void, ::Void, ::Void, ::Base.#pmap, ::WorkerPool, ::#wrapper, ::UnitRange{Int64}) at ./pmap.jl:121
 in pmap(::WorkerPool, ::Function, ::UnitRange{Int64}) at ./pmap.jl:80
 in #pmap#726(::Array{Any,1}, ::Function, ::Function, ::UnitRange{Int64}) at ./pmap.jl:146
 in pmap(::Function, ::UnitRange{Int64}) at ./pmap.jl:146

@amitmurthy
Copy link
Contributor Author

That is a different error since wrapper is not defined on the remote worker. Use @everywhere and you should be good to go.

@everywhere function wrapper(y)
                  c = x->x
                  d = x->c(x)
                  d(y)
       end

@bdeonovic
Copy link
Contributor

I guess what I'm trying to get at here is how does this relate to the error we are having in Mamba (brian-j-smith/Mamba.jl#109)

@amitmurthy
Copy link
Contributor Author

I believe it is the issue of global closures not being serialized which #19594 addresses.

@odow
Copy link
Contributor

odow commented May 9, 2017

Is this failure the local case?

The following works in Julia v0.5.x but is broken in 0.6.0-rc1 and latest 0.7 binaries.

julia> y = (x) -> (() -> x)
(::#1) (generic function with 1 method)

julia> g = y(1)

julia> g()
1

julia> io = IOBuffer()
IOBuffer(data=UInt8[...], readable=true, writable=true, seekable=true, append=fa
lse, size=0, maxsize=Inf, ptr=1, mark=-1)

julia> serialize(io, y)

julia> seekstart(io)
IOBuffer(data=UInt8[...], readable=true, writable=true, seekable=true, append=fa
lse, size=316, maxsize=Inf, ptr=1, mark=-1)

julia> deserialize(io) # crashes
ERROR: type UnionAll has no field name
Stacktrace:
 [1] handle_deserialize(::SerializationState{Base.AbstractIOBuffer{Array{UInt8,1
}}}, ::Int32) at .\serialize.jl:706
 [2] deserialize(::SerializationState{Base.AbstractIOBuffer{Array{UInt8,1}}}) at
 .\serialize.jl:634
 [3] deserialize_expr(::SerializationState{Base.AbstractIOBuffer{Array{UInt8,1}}
}, ::Int64) at .\serialize.jl:881
 [4] handle_deserialize(::SerializationState{Base.AbstractIOBuffer{Array{UInt8,1
}}}, ::Int32) at .\serialize.jl:693
 [5] deserialize(::SerializationState{Base.AbstractIOBuffer{Array{UInt8,1}}}) at
 .\serialize.jl:634
 [6] deserialize_expr(::SerializationState{Base.AbstractIOBuffer{Array{UInt8,1}}
}, ::Int64) at .\serialize.jl:881
 [7] handle_deserialize(::SerializationState{Base.AbstractIOBuffer{Array{UInt8,1
}}}, ::Int32) at .\serialize.jl:693
 [8] deserialize(::SerializationState{Base.AbstractIOBuffer{Array{UInt8,1}}}) at
 .\serialize.jl:634
 [9] deserialize_expr(::SerializationState{Base.AbstractIOBuffer{Array{UInt8,1}}
}, ::Int64) at .\serialize.jl:881
 [10] handle_deserialize(::SerializationState{Base.AbstractIOBuffer{Array{UInt8,
1}}}, ::Int32) at .\serialize.jl:693
 [11] deserialize_array(::SerializationState{Base.AbstractIOBuffer{Array{UInt8,1
}}}) at .\serialize.jl:870
 [12] handle_deserialize(::SerializationState{Base.AbstractIOBuffer{Array{UInt8,
1}}}, ::Int32) at .\serialize.jl:672
 [13] deserialize(::SerializationState{Base.AbstractIOBuffer{Array{UInt8,1}}}, :
:DataType) at .\serialize.jl:1074
 [14] handle_deserialize(::SerializationState{Base.AbstractIOBuffer{Array{UInt8,
1}}}, ::Int32) at .\serialize.jl:722
 [15] deserialize(::SerializationState{Base.AbstractIOBuffer{Array{UInt8,1}}}) a
t .\serialize.jl:634
 [16] deserialize(::SerializationState{Base.AbstractIOBuffer{Array{UInt8,1}}}, :
:Type{Method}) at .\serialize.jl:781
 [17] handle_deserialize(::SerializationState{Base.AbstractIOBuffer{Array{UInt8,
1}}}, ::Int32) at .\serialize.jl:722
 [18] deserialize_array(::SerializationState{Base.AbstractIOBuffer{Array{UInt8,1
}}}) at .\serialize.jl:870
 [19] handle_deserialize(::SerializationState{Base.AbstractIOBuffer{Array{UInt8,
1}}}, ::Int32) at .\serialize.jl:672
 [20] deserialize(::SerializationState{Base.AbstractIOBuffer{Array{UInt8,1}}}) a
t .\serialize.jl:634
 [21] deserialize_typename(::SerializationState{Base.AbstractIOBuffer{Array{UInt
8,1}}}, ::UInt64) at .\serialize.jl:937
 [22] deserialize(::SerializationState{Base.AbstractIOBuffer{Array{UInt8,1}}}, :
:Type{TypeName}) at .\serialize.jl:890
 [23] handle_deserialize(::SerializationState{Base.AbstractIOBuffer{Array{UInt8,
1}}}, ::Int32) at .\serialize.jl:722
 [24] deserialize(::SerializationState{Base.AbstractIOBuffer{Array{UInt8,1}}}) a
t .\serialize.jl:634
 [25] handle_deserialize(::SerializationState{Base.AbstractIOBuffer{Array{UInt8,
1}}}, ::Int32) at .\serialize.jl:678
 [26] deserialize(::SerializationState{Base.AbstractIOBuffer{Array{UInt8,1}}}) a
t .\serialize.jl:634
 [27] handle_deserialize(::SerializationState{Base.AbstractIOBuffer{Array{UInt8,
1}}}, ::Int32) at .\serialize.jl:681
 [28] deserialize(::SerializationState{Base.AbstractIOBuffer{Array{UInt8,1}}}) a
t .\serialize.jl:634
 [29] deserialize(::Base.AbstractIOBuffer{Array{UInt8,1}}) at .\serialize.jl:631
julia> versioninfo()
Julia Version 0.7.0-DEV.67
Commit d214d57* (2017-05-09 15:12 UTC)
Platform Info:
  OS: Windows (x86_64-w64-mingw32)
  CPU: Intel(R) Core(TM) i7-4770 CPU @ 3.40GHz
  WORD_SIZE: 64
  BLAS: libopenblas (USE64BITINT DYNAMIC_ARCH NO_AFFINITY Haswell)
  LAPACK: libopenblas64_
  LIBM: libopenlibm
  LLVM: libLLVM-3.9.1 (ORCJIT, haswell)

@odow odow mentioned this issue May 9, 2017
Closed
@amitmurthy
Copy link
Contributor Author

Probably introduced in #21514 .

cc: @JeffBezanson

@amitmurthy
Copy link
Contributor Author

The original issue has been closed by #19594 . Will open a new issue for #19456 (comment)

@amitmurthy amitmurthy mentioned this issue May 11, 2017
Closed
@mkitti mkitti mentioned this issue Jun 24, 2020
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Indicates an unexpected problem or unintended behavior parallelism Parallel or distributed computation regression Regression in behavior compared to a previous version
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@amitmurthy @bdeonovic @odow