feat(relationship 2.0): use REMOVE_ALL_EDGES_FROM_SOURCE by default for ingesting 2.0 relationships #463
Conversation
…or ingesting 2.0 relationships
| * @throws IllegalArgumentException if the asset urn and the relationship source urn don't match | ||
| */ | ||
| public static <URN, RELATIONSHIP extends RecordTemplate> void validateRelationshipSource(URN assetUrn, RELATIONSHIP relationship) { | ||
| Urn sourceUrn = getSourceUrnFromRelationship(relationship); |
There was a problem hiding this comment.
do we still require the relationship to define the source?
There was a problem hiding this comment.
Removed. the source urn that will be used will be the asset urn during ingestion.
| .map(relationships -> new BaseLocalRelationshipBuilder.LocalRelationshipUpdates( | ||
| relationships, relationships.get(0).getClass(), BaseGraphWriterDAO.RemovalOption.REMOVE_NONE)) | ||
| .map(relationships -> new LocalRelationshipUpdates( | ||
| relationships, relationships.get(0).getClass(), BaseGraphWriterDAO.RemovalOption.REMOVE_ALL_EDGES_FROM_SOURCE)) |
There was a problem hiding this comment.
will this code have any impact to the existing V1 relationships, before they are fully migrated to V2?
There was a problem hiding this comment.
This code is all in the if block
if (_relationshipSource == RelationshipSource.ASPECT_METADATA)
so it will only affect the v2 models. @ybz1013 is working to make a "dual-write" mode which will write both v1 and v2 relationships (into different tables).
There was a problem hiding this comment.
He will also add a validation to make sure that the v1 relationships (derived from relationship builder) == v2 relationships (extracted from aspect) while in dual-write mode so that we can avoid data inconsistencies during the transition phase.
There was a problem hiding this comment.
Sounds good. Thanks for confirming
Summary
In Model 1.0, when new relationships are ingested, existing relationships are handled (i.e. removed) according to user-defined removal options. There are 4 possible options:
Testing Done
adjusted some unit tests
Checklist