Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

profile stitch #444

Merged
merged 10 commits into from
Aug 15, 2018
Merged

profile stitch #444

Show file tree
Hide file tree
Changes from 6 commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
8fd0958
profile stitch
harleensahni Jul 23, 2018
bbc1245
remove payload reference from this branch
harleensahni Jul 24, 2018
934910d
up version in package.json
harleensahni Jul 24, 2018
22529e2
cleanup from code review
harleensahni Jul 27, 2018
3fe5501
clean up from pr
harleensahni Aug 2, 2018
7e81df1
move back to @id where appropriate
harleensahni Aug 2, 2018
a45a89a
Model identities for profile stitching with an array of Identities in…
harleensahni Aug 14, 2018
7f90489
Merge branch 'master' into add-profile-stitch-to-segment-membership
kstreeter Aug 14, 2018
f7fe51f
Merge branch 'master' into add-profile-stitch-to-segment-membership
kstreeter Aug 15, 2018
9018d5a
Merge branch 'master' into add-profile-stitch-to-segment-membership
kstreeter Aug 15, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion package.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@
"aem_user": "packageUser",
"aem_password": "override me securely",
"markdown-importer-version": "0.0.4",
"schemas": 211
"schemas": 213
},
"scripts": {
"clean": "rm -rf docs/reference",
Expand Down
45 changes: 44 additions & 1 deletion schemas/context/experienceevent.example.1.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -114,5 +114,48 @@
},
"xdm:marketing": {
"xdm:trackingCode": "marketingcampaign111"
}
},
"xdm:profileStitching": [
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A couple of questions:

  • What is the use case for adding profileStitching to ExperienceEvent?
  • Is the user expected to populate this via ETL jobs?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To support profile stitching in realtime for use with realtime segmentation
Someone could populate this with an ETL job if they wanted, but that wasn't the use case I was considering. I think it's open enough to support that.

{
"xdm:profileStitchID": {
"@id": "https://data.adobe.io/entities/profileStitchIdentity/1",
"xdm:namespace": {
"xdm:code": "AAM"
}
},
"xdm:version": "1.0",
"xdm:endUserIds": {
"https://ns.adobe.com/experience/mcid": {
"xdm:id": "https://data.adobe.io/entities/identity/92312748749128",
"additionalIDs": [
"https://data.adobe.io/entities/identity/62312748749321",
"https://data.adobe.io/entities/identity/49312748749132"
],
"xdm:namespace": {
"xdm:code": "ECID"
}
}
}
}
],
"xdm:segmentMemberships": [
{
"xdm:segmentID": {
"@id": "https://data.adobe.io/entities/identity/92312748749128",
"xdm:namespace": {
"xdm:code": "AAM"
}
},
"xdm:profileStitchID": {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there a reason to add this to each segment realization and each ExperienceEvent? This information can be looked up on demand from the segment metadata.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, because without it, this event isn't complete. You'd have to reference the segment definition to look up profile stitching that occurred in this event (in ExperienceEvent.profileStitching) to what profiles were used to qualify this segment. That doesn't seem ideal. The alternative to avoid that would be to just embed endUserIds here instead of profileStitchID. That would be much more verbose if multiple segments were realized on this call since the same set of endUserIDs would be repeated in multiple memberships. Also, we may be in a position where we want to record information about profile stitching in ExperienceEvent.profileStitching for other use cases down the road. If that happens, then the same set of information would be in ExperienceEvent.profileStitching.endUserIds and each segmentmembership endUserIds.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@mohrahit, another alternative existed also. That was to have segmentmembership be a property under each profile stitch here in ExperienceEvent. That felt too nested and more importantly, assumed that all segmentation happened with profile stitching.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@mohrahit, are you okay with this approach?

"@id": "https://data.adobe.io/entities/profileStitchIdentity/1",
"xdm:namespace": {
"xdm:code": "AAM"
},
"xdm:lastQualificationTime": "2017-09-26T15:52:25+00:00",
"xdm:version": "1.0",
"xdm:validUntil": "2017-12-26T15:52:25+00:00",
"xdm:status": "realized"
}
}
]
}
25 changes: 25 additions & 0 deletions schemas/context/experienceevent.schema.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -126,6 +126,31 @@
"description": "The media activity information related to the experience event"
}
},
"xdm:profileStitching" : {
"title": "Profile Stitching",
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This seems like a very Adobe-centric thing, yes? If so, it belongs in an extension and not in the primary schema

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think so. It's about which set of profiles segmentation is running against. Adobe has some functionality around this, but I think others in the industry do too.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

well, I would want to understand how it fits into the larger industry context before accepting this in a common schema.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Understood, should we talk offline? Or do you have something in particular you're looking for?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Certainly happy to chat offline, but what I am looking for is either an existing standard for this that we can point to from this common schema...or we move it to an Adobe extension.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@lrosenthol - the concept of merging multiple profile records together to make a golden record is quite common in the industry. The term stitch and merge are synonymous. The Industry term for doing this in old school databases is creating the Golden Record and is a function of Master Data Management. We do it in real time and use simular concepts. Harleens us of the term "profile stitch" in this proposal is understandable by non Adobe practitioners without explanation.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@cdegroot-adobe if I do a web search on "profile stitching", only one of the hits is about this (and it actually calls it visitor stitching).
Is there any existing industry reference document that we can point to that defines this?

"description": "Details about the ids that were joined by profile stitching.",
"$ref": "https://ns.adobe.com/xdm/context/profileStitch"
},
"xdm:segmentMemberships": {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is a "segment", where it is defined, is it specific to an Adobe workflow or generic?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's not defined yet in it's own xdm. This pull request didn't introduce the concept of segment membership, it already exists on profile and the PR for the segment membership is merged. I'm just adding it experience event. It's meant to be generic, but this will definitely be used by Audience Manager and I think Target.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Got it. OK - no issue there.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Issue Closed

"title": "Segment Memberships",
"description": "The segments associated with this experience event",
"type": "array",
"items": {
"type": "object",
"allOf": [
{
"$ref": "https://ns.adobe.com/xdm/context/segmentmembership"
},
{
"properties": {
"xdm:profileStitchID": {
"$ref": "https://ns.adobe.com/xdm/context/profileStitchIdentity"
}
}
}
]
}
},
"required": ["@id", "xdm:timestamp", "xdm:endUserIDs"]
}
},
Expand Down
8 changes: 8 additions & 0 deletions schemas/context/identity.schema.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,14 @@
"type": "string",
"description": "Identity of the consumer in the related namespace."
},
"xdm:additionalIDs": {
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Additional Ids are also needed to support multiple ECID --> CRMID mappings that are provided by co-op and private graph customers. The support for additional Identities should be done outside of the Identity object where each id specified, is of type xdm:Identity. Either in the EndUserIds structure or in a separate Identity extension for ExperiencEvent.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@harleensahni please review with Rahul, I agree on your proposal, this is likely just something you need to talk through.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@biswas-adobe has concerns with Identity itself being modified. It's beginning to look like a better approach to using EndUserID here (as well as in Experience Event) is to go with an array of Identities.

Modeling with additional IDs is not compatible with #459 since it does not allow for different authentication states for each id in a namespace.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@harleensahni I also feel like if would be better to use an array of Identity. This is how the same problem was handled in Profile.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@kstreeter, I can change to array of Identity. The problem is that we will face this same issue on EndUserIDs itself in other use cases though, and I think we're just punting the issue. There are cases where a customer can send multiple ids for the same Identity namespace on the same call. These will need to be recorded on the ExperienceEvent and that uses EndUserID.

It's also weird that we would model Identities in two different ways for anyone who is consuming these events.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@kstreeter @biswas-adobe I've changed this to be an array of Identity

"title": "Additional Identifiers",
"type":"array",
"items": {
"type": "string"
},
"description": "If endUser has more than one identity in the namespace, additional identity ids are listed here."
},
"xdm:namespace": {
"title": "Namespace",
"$ref": "https://ns.adobe.com/xdm/context/namespace",
Expand Down
22 changes: 22 additions & 0 deletions schemas/context/profilestitch.example.1.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
{
"xdm:profileStitchID": {
"@id": "https://data.adobe.io/entities/profileStitchIdentity/1",
"xdm:namespace": {
"xdm:code": "AAM"
}
},
"xdm:version": "1.0",
"xdm:endUserIds": {
"https://ns.adobe.com/experience/mcid": {
"xdm:id": "https://data.adobe.io/entities/identity/92312748749128",
"additionalIDs": [
"https://data.adobe.io/entities/identity/62312748749321",
"https://data.adobe.io/entities/identity/49312748749132"
],
"xdm:namespace": {
"xdm:code": "ECID"
}
}
}
}

40 changes: 40 additions & 0 deletions schemas/context/profilestitch.schema.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
{
"meta:license": [
"Copyright 2018 Adobe Systems Incorporated. All rights reserved.",
"This work is licensed under a Creative Commons Attribution 4.0 International (CC BY 4.0) license",
"you may not use this file except in compliance with the License. You may obtain a copy",
"of the License at https://creativecommons.org/licenses/by/4.0/"
],
"$id": "https://ns.adobe.com/xdm/context/profileStitch",
"$schema": "http://json-schema.org/draft-06/schema#",
"title": "Profile Stitch",
"type": "object",
"description": "Details about the ids that were joined by profile stitching.",
"definitions": {
"profileStitch": {
"properties": {
"xdm:profileStitchID": {
"title": "Profile Stitch ID",
"$ref": "https://ns.adobe.com/xdm/context/profileStitchIdentity",
"description": "The identity of the segment or snapshot definition in with the domain of the specific system that processes that type of segment."
},
"xdm:version": {
"title": "Version",
"type": "string",
"description": "The version of the profile stitch definition used. Useful for debugging."
},
"xdm:endUserIDs": {
"title": "End User IDs",
"$ref": "https://ns.adobe.com/xdm/context/enduserids",
"description": "Condensed, normalized encapsulation of all end user identifiers."
}
}
}
},
"allOf": [
{
"$ref": "#/definitions/profileStitch"
}
],
"meta:status": "experimental"
}
6 changes: 6 additions & 0 deletions schemas/context/profilestitchidentity.example.1.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
{
"@id": "https://data.adobe.io/entities/profileStitchIdentity/1",
"xdm:namespace": {
"xdm:code": "AAM"
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Don't you also need an xdm:xid here? What use is the namespace without the thing you are namespacing?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wouldn't this as-is namespace @id.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@id is the ID of that particular instance of that schema - if that is what you are doing, then yes, I would agree. I thought you were referring to some other object...which would be via xdm:id

}
}
42 changes: 42 additions & 0 deletions schemas/context/profilestitchidentity.schema.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,42 @@
{
"meta:license": [
"Copyright 2018 Adobe Systems Incorporated. All rights reserved.",
"This work is licensed under a Creative Commons Attribution 4.0 International (CC BY 4.0) license",
"you may not use this file except in compliance with the License. You may obtain a copy",
"of the License at https://creativecommons.org/licenses/by/4.0/"
],
"$id": "https://ns.adobe.com/xdm/context/profileStitchIdentity",
"$schema": "http://json-schema.org/draft-06/schema#",
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

also needs @id

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

see earlier comment on it, we will pick up all the @ids in a dedicated PR.

"title": "Identity",
"type": "object",
"description": "ProfileStitchIdentity is used to clearly distinguish Profile Stitching from multiple sources. ProfileStitchIdentity is established by an Profile Stitch identity provider, which itself is referenced in the `namespace` attribute. Within each `namespace`, the profileStitchIdentity is unique.",
"definitions": {
"profileStitchIdentity": {
"properties": {
"@id": {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@id->"xdm:id"

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we want this as @id similar to how we've done with segment identity. We will have profiles stitching defined as full objects in XDM that should be referenced as their own entities

"title": "Identifier",
"type": "string",
"format": "uri",
"description": "Identity of the Profile Stitch in the related namespace."
},
"xdm:namespace": {
"title": "Namespace",
"$ref": "https://ns.adobe.com/xdm/context/namespace",
"description": "The namespace associated with the `xid` attribute."
},
"xdm:xid": {
"title": "Experience Identifier",
"type": "string",
"description": "When present, this value represents a cross-namespace identifier that is unique across all namespace-scoped identifiers in all namespaces."
}
}
}
},
"allOf": [
{
"$ref": "#/definitions/profileStitchIdentity"
}
],
"meta:status": "experimental"
}