-
Notifications
You must be signed in to change notification settings - Fork 2
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Serialization and Marshaling (again) #590
Comments
To answer your earlier question, here is the first issue: julia> bse = BlobStoreEntry(:a, uuid4(), :abc, "", "", "", "", ZonedDateTime("2020-08-11T14:32:27.12+02:00"))
ERROR: ArgumentError: Unable to parse string "2020-08-11T14:32:27.12+02:00" using format dateformat"yyyy-mm-ddTHH:MM:SS.ssszzz". Unable to parse date time. Expected directive DatePart(sss) at char 21
Stacktrace:
[1] macro expansion at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.5/Dates/src/parse.jl:104 [inlined]
[2] tryparsenext_core(::String, ::Int64, ::Int64, ::Dates.DateFormat{Symbol("yyyy-mm-ddTHH:MM:SS.ssszzz"),Tuple{Dates.DatePart{'y'},Dates.Delim{Char,1},Dates.DatePart{'m'},Dates.Delim{Char,1},Dates.DatePart{'d'},Dates.Delim{Char,1},Dates.DatePart{'H'},Dates.Delim{Char,1},Dates.DatePart{'M'},Dates.Delim{Char,1},Dates.DatePart{'S'},Dates.Delim{Char,1},Dates.DatePart{'s'},Dates.DatePart{'z'}}}, ::Bool) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.5/Dates/src/parse.jl:38
[3] macro expansion at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.5/Dates/src/parse.jl:150 [inlined]
[4] tryparsenext_internal at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.5/Dates/src/parse.jl:125 [inlined]
[5] parse at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.5/Dates/src/parse.jl:282 [inlined]
[6] ZonedDateTime(::String, ::Dates.DateFormat{Symbol("yyyy-mm-ddTHH:MM:SS.ssszzz"),Tuple{Dates.DatePart{'y'},Dates.Delim{Char,1},Dates.DatePart{'m'},Dates.Delim{Char,1},Dates.DatePart{'d'},Dates.Delim{Char,1},Dates.DatePart{'H'},Dates.Delim{Char,1},Dates.DatePart{'M'},Dates.Delim{Char,1},Dates.DatePart{'S'},Dates.Delim{Char,1},Dates.DatePart{'s'},Dates.DatePart{'z'}}}) at /home/dehann/.julia/packages/TimeZones/v0mfN/src/parse.jl:88
[7] ZonedDateTime(::String) at /home/dehann/.julia/packages/TimeZones/v0mfN/src/parse.jl:87
[8] top-level scope at REPL[8]:1 EDIT: we can hodge podge this together using something similar to function in #589 |
From SLACK
|
That is a TimeZones.jl bug: JuliaTime/TimeZones.jl#83 This is how I temporary patched it in variable timestamp: But we need a more permanent solution... |
|
|
Technologies to consider: These are two things we need to support in the future:
|
JSON standard bearers
|
|
Javascript is the OG JSON - JSON stands for Java Script Object Notation. So they're arguably the gold standard. |
The three use-cases we need to address when we refactor serialization packing/unpacking are:
|
So next steps:
|
On @Affie's point - we should updated FileDFG so that it writes independent elements to subfolders. It will then resemble the Neo4J structure. This minimizes the stringification of subelements, but it also allows us to easily make a Fuse driver. |
EDIT DF, just for completeness, here is how to save and load with julia's built in serializer. julia> using Serialization
julia> serialize("dfgname", dfgtosave)
julia> loadeddfg = deserialize("dfgname") |
DF EDIT, just to be clear the top level was decided to be JSON, but there is scope to have different internal content according to standardized Hi, just to follow-up on earlier comments. I'm adding these references for context, we have been at this for a while: Existing multilang interfaces motivated JSON over:
in the first place. GraffSDK became the stake in the ground, so I'd like us to stick nearby this decision for a while. Punchline is we are not going back to a world of more than one serialization standard. If something about JSON is slow in Julia here at 3Q20, then that is the way it is. Later, and with more resources, we can worry about ultra-uber-performance on the serialization, but for now we have a hard fought consensus with plain "Julia high-performance" JSON and that should be good enough. Versatility and commonality is much, much, much, more important at this stage; and we need to focus some on good sanitation/robustness of the serialized objects. Minor quality improvements is okay for the short term. What we need now is miles on the most recent code, so we can catalog failures, problems, issues, shortcomings, and unpleasantness: please comment them here in 590 and we can revisit in the medium term.
Also see:
cc @Affie , @jim-hill-r , @GearsAD |
Also keep #668 in mind for future developments |
Given some new code for serializing distributions, I'm thinking we should do away with the DFG assumption that "Packed*Type" exists on the packed type name. Instead, each variable and factor packing and unpacking converter should deal with that, and defiitely store the packed and unpacked as mimeType information inside the serialized object. I.e. the a serialized object MUST have the
That makes the provider of Maybe worth doing this for both Variables and Factors, which means something about the abstraction should be perhaps |
Proposal for In other words, cc @GearsAD , @Affie , @jim-hill-r |
Can use https://github.com/JuliaIO/Tar.jl for multiplatform |
Following fix to JuliaRobotics/IncrementalInference.jl#1155, there is a new type stability issue when using unpack functions. The trouble starts on this line: With the non-concrete A difference in type-stability when building IIF.CCW objects can occur. When building FMd, not all Variable type information is properly included, see: This results in subtle differences of type (and worse an unresolved type-instability any time we use de-serialized DFG objects!). The original dfg object has a hard type while constructing a new ::FactorMetadata{
Vector{DFGVariable{Pose3}},
Vector{Symbol},
Vector{Matrix{Float64}},
Nothing
} But at time of writing, tests in RoME: Results in: ::FactorMetadata{
Vector{DFGVariable},
Vector{Symbol},
Vector{Matrix{Float64}},
Nothing
} notice non-concrete type |
Just linking
|
Just making sure it is captured... With DFG v0.15 we changed point data to be This new serialization is still being fleshed out, so stay tuned. |
When unpacking factors and trying to get the In utopia, deserialization would only build the DFGFactor once (since it is immutable). The current workaround will likely just make copies of the factor object (to change the type of FGC inside CCW, and then return the new copy as early as possible in the deserialization process. |
One of the new tricks for unordered marshaling is to use keyword constructors on the packed type: struct MyType
a
b
end
MyType(;a=1, b="hi") = MyType(a,b) Base already does this in automated way, along with optional values: Base.@kwdef struct MyType
a::Int = 1
b::String
end
mt = MyType(b="bye") Furthermore, with JSON2's use of # NamedTuple gets created somewhere, likely a JSON package reading text
nt = (; a=2, b="again!")
mt = MyType(;nt...) See #848, and JuliaRobotics/IncrementalInference.jl#1476 for early examples. |
Alright, how's this for Unmarshaling with JSON? Also:
EDIT, deleted the copy I have here, and just updating the single comment over by Unmarshal.jl 39. Perhaps the content comes back here if we choose to implement it only here rather than upstream. xref |
After working on CaesarInternal, I believe I have a good handle on this. My recommendation:
Notes:
Advantages:
|
Just capturing ideas for discussion.
History
So JSON
Options
JSON.jl + unmarshal.jl
...
JSON2.jl
Deprecated in favor of JSON3
JSON3.jl
See this for the 2 trait options:
Struct()
orMutable()
In short Struct() is fast, but the order is important (So has to be preserved if created/modfied from outside).
Example:
The text was updated successfully, but these errors were encountered: