Complete understanding pipeline for objects.

[ghost]

So, this is an item to track the progress on the understanding pipeline;
I'm intending to get code to handle the "understanding" of objects written first, as mentioned in the meeting: this will let me fill in the interfaces and make sure that I have all fo the different value types that I intend to support.

The test code for composites / scalars is here:
https://github.com/ecologylab/simplJava/blob/3fc7fe1568a72ab9554ecd3a967fde3613c0f0ac/simplTests/test/simpl/interpretation/UnderstandingCompositesTest.java

The core understander is here:
https://github.com/ecologylab/simplJava/blob/beiber/simplCore/src/simpl/interpretation/SimplUnderstander.java

The interface here:
https://github.com/ecologylab/simplJava/blob/beiber/simplCore/src/simpl/interpretation/SimplInterpretation.java

A scalar interpretation w/ resolution:
https://github.com/ecologylab/simplJava/blob/beiber/simplCore/src/simpl/interpretation/ScalarInterpretation.java

A composite interpretation w/resolution:
https://github.com/ecologylab/simplJava/blob/beiber/simplCore/src/simpl/interpretation/CompositeInterpretation.java

More details to come, but this is what I'm at so far.
@amwebb @andruidk @quyin

[ghost]

WORD OF WARNING: THAR BE VASTLY MINIMAL COMMENTS HERE.

[ghost]

Composites and lists are solid. Working on maps.

There will be some remaining issues w/ collection types that'll need to get cleaned up. I think I'll add a collectioninterpretation for collections (Since lists have ordering guarantees that require distinctly different strategies for handling them; already taken care of w/ listinterpretation) This will bring collection support up to language parity.

Some other secondary issues w/ handling implementation creation of lists / collections / maps are going to exist, but that is primarily related to the gaps in description pipeline, addressed in issues #76 , #77

[quyin]

so this is a layer that handles conversion between objects and
serialization forms, correct? I'm trying to understand how this works with
de/serialization conceptually. the descriptors themselves do not (and they
should not) handle this conversion directly, correct?

Best Regards,
Yin Qu (屈垠)

On Sun, Feb 24, 2013 at 5:46 PM, Tom White notifications@github.com wrote:

Composites and lists are solid. Working on maps.

There will be some remaining issues w/ collection types that'll need to
get cleaned up. I think I'll add a collectioninterpretation for collections
(Since lists have ordering guarantees that require distinctly different
strategies for handling them; already taken care of w/ listinterpretation)
This will bring collection support up to language parity.

Some other secondary issues w/ handling implementation creation of lists /
collections / maps are going to exist, but that is primarily related to the
gaps in description pipeline, addressed in issues #76#76,
#77 #77

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/74#issuecomment-14019173.

[ghost]

so this is a layer that handles conversion between objects and
serialization forms, correct?

Yes. "Interpretation / Understanding" are the format agnostic pieces on the outside of the process, format specific stuf fhappens on the mapping from Interpretation -> Format text and Format text -> Interpretation.

the descriptors themselves do not (and they
should not) handle this conversion directly, correct?

Correct.

Let me explain by outlining the general flow for a scalar object:

class myObject
{ 
@simpl_scalar public Integer myInt; 
// constructor etc ellided
}

let's say I have an instance of myObject { myInt = 13 }...

An "Interpretation" of myObject has the tag name of the root object and the interpretation of the scalar field:
Represented as a stack of interpretations: (An interp on each line)

my_object
myInt = "13" (IntegerType)  

Note, the integer value of 13 is marshalled via the IntegerType to the string "13" The format doesn't need to know about the mapping from 13->"13" , just that the end result is "13"

Format specific encoding happens at the format level, strings are sent to formats in interpretations unescaped and those formats perform escaping.

To serialize this to a format, I convert this interpretation (or any series of interpretations) to the requisite format.
So a scalar mapping for a scalar interpretation in json could look like fieldname="fieldvalue" (the actual value of fieldValue in this example would really be json_escape_somehow(interpretation's field value) but at any rate...

This gets converted pretty easily to JSON / xml / whatever

{my_object:{ myInt="13"}}

or maybe

<my_object myInt ="13" />

(So, we've serialized data from instance to interpretation to format. )

Let's deserialize!

First, we convert the format to interpretation, guided by the typescope, etc.

Each format has its own mapping from format specific text -> Interpretations.

Both formats will still ultimately come back to the following representation in interpretations: (I think the scalartype information will actually get rehydrated in the understanding code, but that's a specific detail that I'm still going back and forth on; the info is already there in the ts, just a matter of placing it)

my_object
myInt = "13" (IntegerType)  

These then get sent to the understander. It goes through and resolves the interpretations.

So, it creates an instance of my_object (myObject)

myObject {}

it then resolves() the scalar interpretation myInt = "13" (IntegerType) -> The string value of "13" is sent to IntegerType's setField... with the field myInt on myObject. (This allows it to resolve the primitive types as well. see also: earlier fixes in this space.

So, after resolving that interp, we have myObject {myInt =13} again. Ad we're happy because we've roundtripped successfully.

to explain the composite, list, collection, and map handling, we'll take a brief detour through the SimplInterpretation interface and then after that it'll all be clear

"But Tom," you may say "How does this notion of 'resolving' an interpretation on a specific field work when I'm working with lists and maps and composites and all of the things?!'

public interface SimplInterpretation {
    void resolve(Object context, Set<String> refSet, UnderstandingContext understandingContext)
            throws SIMPLTranslationException;

    Object getValue(Object context, Set<String> refSet, UnderstandingContext understandingContext)
                         throws SIMPLTranslationException;
}

Any interpretation can do things: It can "resolve" or it can "getValue"
When it resolves, it places a value inside a field in a context object. (So, when the value of "13" for an IntegerType gets placed into the field myInt, it's getting resolved.)

When it "getsValue", it merely gets the value that corresponds to the interpretation.

So: Say I have a list of scalars, [0,1,2,3](maybe it's List<Integer> myList in some class.)

I represent that as a "list interpretation" which contains a list of "simplinterpretations"

In this specific case: the interpretations inside the list interp are scalar interpretations that represent the numbers 0,1,2,3. Using a similar notation from earlier:

myList {
=0 (IntegerType)
=1 (IntegerType)
=2 (IntegerType)
=3 (IntegerType) 
}

I can't "resolve" any of the scalar interpretations, that doesn't make sense because there's no field they resolve to. (the empty string in front of = is intentional. ;) ) However, I can getValue of the interpretation, and then add that to a list object. =0 (IntegerType) => thus becomes the integer value 0

I'm going to explain composites in another comment, gimme a minute. :D

[ghost]

Composites are a list of simpl interpretations, and potentially a REF or an ID.

(I'm going to make some references to existing test code. You can peruse it all here: https://github.com/ecologylab/simplJava/blob/beiber/simplTests/test/simpl/interpretation/UnderstandingCompositesTest.java )

Heres some java code that creates an instance of a composite (this one won't have a reference)

plainComposite pc = new plainComposite();

myScalars orig = new myScalars();
orig.aDouble = 1.3;
orig.aInteger = 13;
orig.aField = "string";

pc.myComposite = orig;
pc.myString = "string";

In our fancy interpretation notation, it may look like this:

plain_composite
myString = "string" (StringType) 
myComposite: my_scalars { 
aDouble = "1.3" (DoubleType)
aInteger = "13" (IntegerType) 
aField = "string" (StringType)
}

myComposite refers to a CompositeInterpretation for a composite type with a tag name my_scalars that happens to be composed of the interpretatiosn for the values of its aDouble, aInteger, aField fields (To restate that)

Understanding this interpretation is straightforward:

1. Create an instance of plain_composite
2. set plain_composite.myString to "string"
3. Create an instance of my_scalars
4. Set its fields w/ the same scalar interp infrastructure used earlier
5. set plain_composite.myComposite to that value.
6. Done!

[ghost]

"So, how does this handle cycles / references?"

So, let's say I have a tree class w/ right and left:

class treeClass 
{
@simpl_composite
myScalars rightNode; //same myScalars from earlier. :) 
@simpl_composite
myScalars leftNode;
}

So, suppose I have an instance of treeClass where leftNode and rightNode are the same....

at "Interpretation time" (the mapping from instance -> interpretations) the interpreter picks one object to give an ID, and the other gets a REF.

My interpretations may look like:

tree_class
leftNode: my_scalars (REF = 1) {}
rightNode: my_scalars (ID = 1) {
aDouble = "1.3" (DoubleType)
aInteger = "13" (IntegerType) 
aField = "string" (StringType)
}

These, upon interpretation -> format, get converted to the format specific representations of ID / REF. (simpl:ref / simpl:id in XML, for example)

On format -> interpretation, we get back the same stack of interpretations...

At "understanding time"... When we first stumble upon a REF, if we haven't seen it before, we add the REF to the refSet and create an instance of that object (which we register in the understandingContext) (We also create an instance and register it in the understandingContext when we see the ID first)

When we resolve whichever composite interpretation contains the ID (And also all of the inner interpretations), we update that object we created, and we get the updates automatically everywhere else we made the REF. (And we remove the entry from the refSet)

If, when we resolve the set of interpretations and we find that the refSet is not empty, we had too many REFS and not enough ID's, so we throw an exception!

Again: Step by step:

1. Create tree_class instance
2. REF 1? Not in the understandingContext!
   • add 1 to the refSet
   • Create instance of my_scalars
   • Set value of tree_class.leftNode to this instance
3. ID 1? This is in the understandingContext!
   • Get that registered instance of my_scalars
   • remove 1 from the refSet
   • Resolve all of the inner simplInterpretations
   • (Set the values just like we set my_scalars in the previous example, that is)
   • Set value of tree_class.rightNode to this instance
4. We're done resolving... is ref set Empty? Yes! We're done!

[ghost]

Underview: What are the benefits of this?

1. Polymorphic interpretation classes are much simpler, allow for greater modularity of our type interpretation / understanding. Implementing new types is pretty pain free, although it can be easy to make a mistake at getValue / resolve distinction at first.
2. Polymorphism wins us some really nice properties in that once composite interpretation is written, anything else can use composites pretty pain free. Once I have a list that can resolve and getValue, I can add list values to list values of map values of composites. Or anything, really.
3. Radically simplifies graph resolution. It's one pass (essentially) with a very nice mapping from serialized representation to the interpretation to the deserialized object.

[ghost]

Underview part 2:
If you want JSON sooner, I"d urge you to hop on the beiber branch and write some tests to try and break the pipeline. I have some tests that I haven't implemeted yet which you could start on, or you could conspire to break it in other ways.

You could also start writing some code to translate a list of interpretations into JSON, or JSON into some interpretations.

[ghost]

Just need to write interpretation for maps and lists, this will borrow the general structure of the composite interp.

[ghost]

Lists: ec8a1fd