How can Thema be used for transformations between types?

[verdverm]

An example context for this pondering

• My side (A) has internal data model that we use for our APIs
• We call (B), an external service, which also has evolving APIs
• We have transformations between the two systems, for example to reshape to

The first question is, in addition to our data models, can we use Thema for the transformations?

It also seems an extra dimension may be added. Consider

• The transformation can be independently changed (we want a different set of fields from the 3rd party source, but our internal types have not changed, nor theirs)
• If either our side (A) or the 3rd party side (B) changes their data model, then more than likely the transformation also needs to be changed. This means both model and transformation are likely to have an amount of mutual versioning, but Thema could help here too?

[sdboyer]

Probably yes, but:

for example to reshape to

Missed the example, and i can imagine a few things coming after "reshape," possible ones here :)

In general though, i would imagine a transform as being a separate kind of thing than a lineage. If we think of the universe of all possible schemas as a graph where schemas are vertices and the known relations between the schemas are edges, then lineages are subgraphs defined by a particular pattern of edges linking them - they provide the bidirectional concreteness preservation invariant.

In such a graph, it sounds like the transformations you're describing are edges between lineage subgraphs that provide unidirectional concreteness preservation between exactly one schema in A and exactly one schema in B - e.g. B@1.2->A@0.4. It seems to me that this should be sufficient - you can:

• Translate an input instance of B to 1.2, whatever its initial version was
• Transform the result to A@0.4 using your edge
• Translate the result to whatever version of A your program wants

And accumulate lacunas from across the process.

So the common useful property here would probably be concreteness preservation (i really have to get that writeup done, sorry, coming soon 😢), but because these transforms are not themselves, defining the schema over which concreteness preservation would be checked, they need not be full-blown lineages. Which is good!

So, to these:

The transformation can be independently changed (we want a different set of fields from the 3rd party source, but our internal types have not changed, nor theirs)

If two transforms differ not in the versions of A and B they target, but in the set of fields being related, then perhaps something meaningfully different is being captured, and they should be considered distinct things with different names?

If not, and there is cause for a program to need to be able to utilize both old and new versions of the transforms, then that would be a case for an internally versioned structure, lineage-style. Not sure what properties could be checked to grant it meaning like a syntactic version, though.

If either our side (A) or the 3rd party side (B) changes their data model, then more than likely the transformation also needs to be changed. This means both model and transformation are likely to have an amount of mutual versioning, but Thema could help here too?

Either A or B evolving is the expectation of Thema, so i think you'd do the three-step operation described above.