Support backlink attributes via the importer #109
Conversation
We've renamed a few types to be 'engine generic' API types, so we need to reflect that into the importer
We're introducing backlink attributes, which allow you to say 'find all the features that this team owns' or similar, based on another catalog type. This is challenging for the importer as this means attributes in an importer config can reference each other. e.g. if I'm creating both Teams and Features in my catalog, I need to wait until Features exists before creating the backlink attribute in Teams. To resolve this, we split the UpdateTypeSchema process into two parts: 1. Create/Update all the types and their schemas 2. Create any new backlinks that don't exist yet (as the other attribute must now be present). Note that to make the code sane, I've pulled the 'if dryRun' up one level as dryRun doesn't have any of these concerns.
paprikati
force-pushed
the
lisa/cat-135-support-backlinks-in-the-catalog-importer
branch
from
0a508b5
to
3c25f84
Compare
|
I think we probably shouldn't merge this until we're about to release the feature, just to avoid any confusion! |
paprikati
force-pushed
the
lisa/cat-135-support-backlinks-in-the-catalog-importer
branch
from
16a1e98
to
c620d6a
Compare
| }) | ||
|
|
||
| if !inCurrentSchema { | ||
| hasNewBacklinks = true |
There was a problem hiding this comment.
You can break here right - you don't need to check the rest of the attributes
| // Update all the type schemas except for new backlinks, which could reference | ||
| // attributes that don't exist yet. | ||
| catalogTypeVersions := map[string]int64{} | ||
| for _, outputType := range cfg.Outputs() { |
There was a problem hiding this comment.
I don't think this is a performance problem that needs to be fixed, but we're going through the types twice - once to figure out what's a new backlink, and once to figure out what isn't.
Using lo.PartitionBy might be useful here to do this in one go, then we only have to implement one bit of logic that works out if an attribute is a new backlink or not.
There was a problem hiding this comment.
Yeah I felt that writing the code like that was kinda more confusing, and given #attributes should be tiny, I think this is fine?
Main thing that would change my mind if it this was confusing to you as a reader
There was a problem hiding this comment.
It was a bit confusing to me while reading it, and I think PartitionBy would be more understandable. But also having read the docs for PartitionBy, I think its behaviour is weird and it'd be a pain to use.
paprikati
deleted the
lisa/cat-135-support-backlinks-in-the-catalog-importer
branch
We're introducing backlink attributes, which allow you to say 'find all the features that this team owns' or similar, based on another catalog type.
This is challenging for the importer as this means attributes in an importer config can reference each other. e.g. if I'm creating both Teams and Features in my catalog, I need to wait until Features exists before creating the backlink attribute in Teams.
To resolve this, we split the UpdateTypeSchema process into two parts:
Note that to make the code sane, I've pulled the 'if dryRun' up one level as dryRun doesn't have any of these concerns.
This will fail with a 422 until we release the feature and flip the feature flag.