Draft Proposed Structure [WIP]

Copyright NavAbility (2020) - All rights reserved.

DFG

Entities, constructors and Accessors

• Should only depend on lower down structs
• Should be testable as units

Abstract DFG

• Struct
• Constructors
• Accessors

Variable

• Struct
• Constructors
• Accessors

VND

• Struct
• Constructors
• Accessors

PPE

• Struct
• Constructors
• Accessors

BigData

• Struct
• Constructors
• Accessors

Factor

• Struct
• Constructors
• Accessors

GenericFunctionNodeData

• Struct
• Constructors
• Accessors

Services

• CRUD, Set opperations, lists, sort, etc.
• The basic API recomended for most users

Abstract DFG

Variable

• CRUD/SET, eg. get/set/update/list/merge
• wrappers of the form verb...(fg, key, [value]) = verb...(getVariable(fg,key), [value])

Tags

• CRUD/SET, eg. get/set/update/list/merge

VND (VariableSolverData)

• CRUD/SET, eg. get/set/update/list/merge

PPE

• CRUD/SET, eg. get/set/update/list/merge

BigData

• CRUD/SET, eg. get/set/update/list/merge

Factor

common//helpers

• additional functions, exist, isVariable

compare

• Generated and standard compare functions
• to be overwritten if needed.

Deprecated

• if possible use @deprecate and add comment of what version.

Drivers

---

[WIP]

• SC - I'm going to move this to docs, using this as a place to formulate thoughts.
  • 🔴 DF, how about keeping this in a Tech Dev Wiki alongside API Spec Wiki?

DFG Design

Couple notes:

• Things that are dictionaries (e.g. variables/factors, estimates) I'm just assuming we're using add/update/delete/list. Similar to LightGraphs API.
  • 🔴 DF, i suggest we follow verbNoun[Special] for all functions.
  • Yes, it would just be 4 verbs - add/update/delete/get.
• For lists (e.g. tags) we just use the same, except there is no update method. You can add and delete from the list. An optional extra is to include an update that actually sets the list with the given items.
  • 🔴 DF, suggest we follow verbNoun[Special] for all functions.
  • SC, Yes, it would just be 4 verbs - add/delete/get.
• 🔴 DF, while the design and API language are related, the API is a human consideration while the data design etc is a technology consideration.
  • SC, not sure what you mean by this, could you clarify?

Terminology

• Primary structures - these are the root structures in the factor graph:
  • Variables: SkeletonDFGVariable, DFGVariableSummary, DFGVariable
  • Factors: SkeletonDFGFactor, DFGFactorSummary, DFGFactor
  • 🔴 DF, i see Variable and Factors as Nouns from an API language perspective.

General API Philosophy

• Everything is passed by reference, unless it cannot be (e.g a DFGVariable from CloudDFG)
  • The effect of this is if I update solver data, or a property on a returned DFGVariable, it will update directly in the graph. The CloudGraphs driver is exception.
  • Another effect of this is that as a user I don't need set operators. E.g. I can work with DFGVariable.tags as I like, and the set operators are just for convenience (the graph is updated in place and I don't need to explicitly set the variable back into the graph for the tag update to be applied). An exception to this is label and timestamp as they are first-class citizens of the variable - they require that a new variable or factor is created to perform the update.
• Increasing levels provide increasing levels of detail (e.g. for a variable - level 0 = DFGSkeleton = label+tag. Level 4 = everything and big data).
  • 🔴 DF, DataLevel1 is not a replacement for DataLevel2. DataLevel1 exists as a convenience for users that do not want to engage with solver details, but it is not a replacement to hide the full marginal belief estimate at the heart of the entire Caesar.jl ecosystem.
• Primary structures (i.e. DFGVariable/DFGFactor) are immutable (I'm still not convinced this is necessary for performance, but going with it for now [would like to see a performance test]. It does help with good practices though)
  • The effect of this is properties on first-class structure (label, timestamp) cannot be updated in place and require the variable is removed/re-added for the update to work
    • 🔴 DF, adjacent to this discussion of how to use these fields is the design which are either mutable or immutable. There are strong reasons for working with immutable structures, but part of the long term strategy not immediately required.
    • SC - Sure, I trust that you have read more on it and I see other benefits, but discussions about performance are very general, e.g. above article says "It can be more efficient."
• Get operators (getters) generally exist for two reasons - either the property does not exist in the inner structure (e.g. getEstimateDict) or we are concerned the property name may change.
• Set operators (setters) generally exist for three reasons - either the structure is immutable (e.g. timestamp and label), there is logic underneath the set, or you have a copy of the datastructure and want to set it back in the graph.
  • 🔴 DF, as above, setters are a clear example of tension between Design and User API -- suggestion is to FIRST ratify the verbNoun definitions BEFORE trying to resolve both design and API challenges in a single discussion -- I strongly feel that the entire API should not follow from a design issue on setters. In other words, we need to be precise about what add/append/set/merge/union/push/put means as a verb, which (i hope) will then result in us writing more than one function as a "setter". I feel the word "setter" might over-simplify all issues relating to Design and API. Julia itself is also finding it's way through this API business -- eg. push[!] on data containers like Array/CircularBuffer, but put! on others like IOBuffer/Channel.
• We generally recommend using the getters and setters if they exist.
  • 🔴 DF, i recommend users refer to the API definitions to find the right* setter and see which options exist in different situations: add/add!/push/push!/union/merge/set/put.
  • SC - Sure, so to address that here: I'm suggesting that we have 4 CRUD verbs (add, get (or list), update, delete) and a few periphery ones (draw, etc.). I don't see a need for more, honestly, at least for general graph operations (adding estimates, solves, etc.). We may need set operators (union, intersect, etc.) if we are combining graphs.

Operations on the Graph

The graph currently works on a CRUD model - addVariable!, getVariable, updateVariable!, deleteVariable!, ls.

• They return either the updated/retrieve structure, nothing if it cannot be found, or an exception if there was an underlying error (e.g. communication failure).
• add* will add it if it doesn't exist and return nothing if it already exists (strict add)
• update* will update an existing entry, and add if it doesn't exist (permissive update)
  • As far as I can tell this would be best to do a merge update for all fields in the object (e.g. tags, solverDataDict, etc.) so that you can use it in conjunction with getVariable(dfg, :a, :parametric) to work only against the parametric solver dictionary. I know we called this mergeUpdate, but it's overly wordy so as long as the functionality is understood, we can just say update is a merge update.
  • To separate the functionality, I suggest a setVariable() that does a complete overwrite.
• getVariable or getFactor have the option to get a specific solver key, which will return only that subset of solver data and estimates.

Operations on the Graph Elements

Primary structure for all nodes:

• label and timestamp (immutable properties)
  • getLabel(variable/factor) and getTimestamp(variable/factor) both just return x.label and x.timestamp respectively
  • setLabel(dfg, variable/factor, newlabel) and setTimestamp(dfg, variable/factor, newtimestamp) both replace the complete variable or factor in the graph and return the new element.
• solvable
  • getSolvable(variable/factor) and setSolvable(variable/factor) get/update a mutable dictionary of node properties.
• tags (mutable list property for Summary and Variable types, not skeleton) [TBD - this is technically a list, so little strange]
  • getTags(variable/factor) returns element.tags.
  • addTags(variable/factor, tags) adds the tags if they don't exist (permissive add)
  • deleteTags(variable/factor, tags removes the tags if they exist (permissive delete)
  • setTags(variable/factor, tags) sets the tags to the provided list [optional method, maybe for backward compatibility]

Primary structure for variables:

• [TBD] getSofttype(dfg, variable) returns the softtype (<: InferenceVariable) for the variable

Primary structure for factors:

• None

This is all the primary elements of all nodes. Beyond this are properties of each variable or factor:

Estimate Data

Dictionary with operators. Update is permissive (updates what it has, i.e. merge).

Solver Data

Dictionary with operators. Update is permissive (updates what it has, i.e. merge).

Big Data

Dictionary with operators. Update is permissive (updates what it has, i.e. merge).

Small Data

Dictionary with operators. Update is permissive (updates what it has, i.e. merge).

Functions

Functions Common to all Graph Types:

• Existence: exists(dfg, Union{Symbol, DFGNode})::Bool
• Connectivity checks: isFullyConnected(dfg)::Bool and hasOrphans(dfg)::Bool

Find Functions

• findClosestTimestamp
  • 🔴 DF, if we follow verbNoun[Adjective] closely this becomes findTimestampClosest, although its probably not serious to define both in particular cases like this?
• findVariableNearTimestamp
  • 🔴 DF, if we follow verbNounNouNAdjective closely this becomes findVariableTimestampNear ...