profile stitch #444

harleensahni · 2018-07-24T12:42:10Z

This PR addresses issue #468.

Add a concept of profile stitch / merge rules to experience event. I had to introduce a concept of profile stitch identity similar to what we have for segment identity, but I'd prefer if we just took a much simpler approach with segment and profile ids as simple strings.

I also add segment membership to experience event and modify identity to support multiple ids for the same namespace (which is a hard requirement for merging).

cdegroot-adobe

Still processing this PR, just made a few small syntactic change requests.

cdegroot-adobe · 2018-07-24T14:08:24Z

schemas/context/identity.schema.json

@@ -20,6 +20,14 @@
          "type": "string",
          "description": "Identity of the consumer in the related namespace."
        },
+        "xdm:additionalIds": {


additionalIDs

cdegroot-adobe · 2018-07-24T14:10:11Z

schemas/context/identity.schema.json

+          "items": {
+            "type": "string"
+          },
+          "description": "If consumer has more than one of the given id, additional ids are listed here."


If endUser has more than one identity in the namespace, additional identity ids are listed here.

cdegroot-adobe · 2018-07-24T14:10:49Z

schemas/context/profileStitch.example.1.json

+  "xdm:version": "1.0",
+  "xdm:endUserIds": {
+    "https://ns.adobe.com/experience/mcid": {
+      "@id": "https://data.adobe.io/entities/identity/92312748749128",


this has now changed to "xdm:id" as of 0.9.3

cdegroot-adobe · 2018-07-24T14:11:03Z

schemas/context/profileStitch.example.1.json

+  "xdm:endUserIds": {
+    "https://ns.adobe.com/experience/mcid": {
+      "@id": "https://data.adobe.io/entities/identity/92312748749128",
+      "additionalIds": [


additionalIDs

cdegroot-adobe · 2018-07-24T14:12:06Z

schemas/context/profilestitchidentity.schema.json

+    "definitions": {
+        "profileStitchIdentity": {
+            "properties": {
+                "@id": {


@id->"xdm:id"

I think we want this as @id similar to how we've done with segment identity. We will have profiles stitching defined as full objects in XDM that should be referenced as their own entities

cdegroot-adobe · 2018-07-24T14:12:24Z

schemas/context/profilestitchidentity.schema.json

+        }
+    ],
+    "meta:status": "experimental"
+}


add newline please.

cdegroot-adobe · 2018-07-24T14:18:23Z

@harleensahni - I really like the additionalIDs. To simplify things I recommend that you pit that in a separate PR as that is quite a distinct thing and we should be able to get closure on it quickly.

I do not quite understand the concept of the core of this addition to ExperienceEvent. I am assuming that the merge rules and profile stitching are added during data-collection. So this is really a log of that. Can you explain the interaction flow a bit please?

lrosenthol · 2018-07-24T14:36:48Z

schemas/context/experienceevent.schema.json

@@ -126,6 +126,31 @@
          "description": "The media activity information related to the experience event"
        }
      },
+      "xdm:profileStitching" : {
+        "title": "Profile Stitching",


This seems like a very Adobe-centric thing, yes? If so, it belongs in an extension and not in the primary schema

I don't think so. It's about which set of profiles segmentation is running against. Adobe has some functionality around this, but I think others in the industry do too.

well, I would want to understand how it fits into the larger industry context before accepting this in a common schema.

Understood, should we talk offline? Or do you have something in particular you're looking for?

Certainly happy to chat offline, but what I am looking for is either an existing standard for this that we can point to from this common schema...or we move it to an Adobe extension.

@lrosenthol - the concept of merging multiple profile records together to make a golden record is quite common in the industry. The term stitch and merge are synonymous. The Industry term for doing this in old school databases is creating the Golden Record and is a function of Master Data Management. We do it in real time and use simular concepts. Harleens us of the term "profile stitch" in this proposal is understandable by non Adobe practitioners without explanation.

@cdegroot-adobe if I do a web search on "profile stitching", only one of the hits is about this (and it actually calls it visitor stitching).
Is there any existing industry reference document that we can point to that defines this?

lrosenthol · 2018-07-24T14:37:12Z

schemas/context/experienceevent.schema.json

+        "description": "Details about the ids that were joined by profile stitching.",
+        "$ref": "https://ns.adobe.com/xdm/context/profileStitch"
+      },
+      "xdm:segmentMemberships": {


What is a "segment", where it is defined, is it specific to an Adobe workflow or generic?

It's not defined yet in it's own xdm. This pull request didn't introduce the concept of segment membership, it already exists on profile and the PR for the segment membership is merged. I'm just adding it experience event. It's meant to be generic, but this will definitely be used by Audience Manager and I think Target.

Got it. OK - no issue there.

Issue Closed

lrosenthol · 2018-07-24T14:39:11Z

schemas/context/profileStitch.schema.json

+    "you may not use this file except in compliance with the License. You may obtain a copy",
+    "of the License at https://creativecommons.org/licenses/by/4.0/"
+  ],
+  "$id": "https://ns.adobe.com/xdm/context/profileStitch",


Also needs @id, as per new plans

Agreed, we need to get that approved first and then we will do the needful. I would like Kevin to return for the @id decision and exactly how we roll it out. In the XC we cannot automatically add them, so we need to do something to limit the explosion of fields in our operational store.

lrosenthol · 2018-07-24T14:39:33Z

schemas/context/profileStitch.schema.json

+  "$schema": "http://json-schema.org/draft-06/schema#",
+  "title": "Profile Stitch",
+  "type": "object",
+  "description": "Details about the ids that were joined by profile stitching.",


This seems like a very Adobe-centric thing, yes? If so, it belongs in an extension and not in the primary schema

Please see earlier comments on this. I am confident this is not an Adobe-centric things at all.

lrosenthol · 2018-07-24T14:40:23Z

schemas/context/profilestitchidentity.schema.json

+        "of the License at https://creativecommons.org/licenses/by/4.0/"
+    ],
+    "$id": "https://ns.adobe.com/xdm/context/profileStitchIdentity",
+    "$schema": "http://json-schema.org/draft-06/schema#",


also needs @id

see earlier comment on it, we will pick up all the @ids in a dedicated PR.

lrosenthol · 2018-07-24T14:42:17Z

schemas/context/profilestitchidentity.example.1.json

+{
+  "@id": "https://data.adobe.io/entities/profileStitchIdentity/1",
+  "xdm:namespace": {
+    "xdm:code": "AAM"


Don't you also need an xdm:xid here? What use is the namespace without the thing you are namespacing?

Wouldn't this as-is namespace @id.

@id is the ID of that particular instance of that schema - if that is what you are doing, then yes, I would agree. I thought you were referring to some other object...which would be via xdm:id

cdegroot-adobe · 2018-07-24T14:46:30Z

schemas/context/profileStitch.schema.json

@@ -0,0 +1,40 @@
+{


Lower case the name of this schema, and the example too please.

harleensahni · 2018-07-26T14:23:48Z

@cdegroot-adobe For splitting into two different PRs, one for addiitonalIDs and one for profile stitching, is it necessary? I ask because the profile stitching PR is useless without the additionalIDs and I want to call out as part of the PR for profile stitching that we'll be using additiionalIDs to have multiple ids for the same namespace.

As far as the question about what this PR does with profile stitch / merge rule and the flow:

Merge rules / profile stitch are defined elsewhere (not addressed in this PR really, apart from declaring the ID format for those). This is similar to the punting approach we've taken with segment memberships but not defining segment yet.
What is recorded in the Experience Event here is the result of a specific invocation of a profile stitch / merge rule that was done to connect this event and the ids in it to other profiles. Basically, what's recorded is that profile stitching that was done in this way (profileStichId, [linkedid_1, linkedid_2, linkedid_n]). And then adding segment membership on experience event with an additional field to refer back to the profile stitch id to find what other profiles that segment realization applies to (besides the one that came in on the event).

lrosenthol · 2018-07-26T16:51:46Z

@harleensahni what do you mean by "multiple IDs for the same namespace"? That just seems wrong.

As long as the "merge rules" you describe could be implemented by anyone else using the same schemas (but perhaps with their own ID format) - that sounds reasonable. If this is Adobe-centric, it needs to move to an extension.

harleensahni · 2018-07-26T18:20:38Z

@lrosenthol

What do you mean by "multiple IDs for the same namespace"? That just seems wrong.

I think it'll be helpful if I give an example. Here the namespace will be ECID, and id will the cookie associated with the ECID and well us two values, ECID_1 (for device 1) and ECID_2 (for device 2). There'll be another namespace for identity of CRMID (possibly based off a hash of customer's email). For this example, we'll have one CRM IDs, CRMID_1.

Let's say that we've received one experience events in the past, EE_1 . On EE_1, we saw that the event came from ECID_1 and the user was logged in with CRMID_1. Now we have another experience event EE_2, the event came from ECID_2 and the user logged in was the same user with CRMID_1. If we have a profile stitching that's operating where it can join ECIDs based on the logged in CRMID, you will have two ECIDs of value ECID_1 and ECID_2 in the same namespace. This is recording that.

As far as what joining those two ECIDs, one thing is segmentation that operates against the joint profiles of ECID_1 and ECID_2. It's important to know which profiles were joined and to have that recorded so that you can record the segmentation back on both ECID_1 and ECID_2.

lrosenthol · 2018-07-26T19:15:10Z

ECID isn't a real namespace - at least as far as XDM is concerned. It's something specific to how you define the string that represents your IDs, yes? Since IDs are just arbitrary strings - that seems out of scope for XDM proper since it doesn't care what you stick in there...

harleensahni · 2018-07-26T19:21:42Z

It's defined in XDM (under it's old name of MCID): https://github.com/adobe/xdm/blob/master/schemas/context/enduserids.schema.json#L20
When I'm saying namespace, I mean the identity space for end user ids.

harleensahni · 2018-07-26T19:23:16Z

The alternative would be to not represent this as EndUserId schema and instead just have an array of Identity. I'm fine with either approach, but we'll need to add multiple id support for end user id either way because that is what Experience Event uses to record the IDs that came in on the call, and it does sometime happen you get multiple ids for the same thing.

lrosenthol · 2018-07-26T19:29:29Z

Gotcha - thanks for that link! (just filed an issue against that schema, #449 , but that's separate)

OK - so if you need multiple items, use an array. What am I missing??

cdegroot-adobe · 2018-07-26T19:39:25Z

@lrosenthol the design of the endUserIDs is very carefully thought out. Having specific properties in known locations is very important for efficiency when there are very large numbers of records, as is the case with ExperienceEvent. That is why we have this design and it is predicated on multiple identities being an edge case. Changing how endUserIDs implements this is not and option. The additionalIDs is a way of catering for the edge case and is fine because they are not used in the same way as the "xdm:id" for the identity. additionalIDs is an array, so covers this case. As I say this specific area is quite carefully curated and what Harleen is doing here is the approach that multiple teams have discussed and come to agreement on.

harleensahni · 2018-07-26T19:43:23Z

The reason I chose to model this using EndUserId here is that is what ExperienceEvent uses for the primary ids that come in with the event here: https://github.com/adobe/xdm/blob/master/schemas/context/experienceevent.schema.json#L43. So I want this to be consistent with that. I think EndUserId was modeled the way it is to support being able to do lookups based on a namespace very quickly since it'll be its own column on disk. It would be weird if I modeled this as an array of EndUserIds since you could have multiple ids under the same namespace in separate instances of EndUserIds (and you can have multiple different namespaces in EndUserIds).

By adding the array of additionalIDs to EndUserIDs, you may not be able to quickly find if a particular id is in a namespace since it might be in the array, but you can at least still tell very quickly if an event has ANY ID in a particular namespace, which is valuable.

lrosenthol · 2018-07-26T19:44:23Z

Well, personally, I am not thrilled with this design but I will accept that we can't change the model at this time.

biswas-adobe · 2018-08-02T01:53:19Z

schemas/context/identity.schema.json

@@ -20,6 +20,14 @@
          "type": "string",
          "description": "Identity of the consumer in the related namespace."
        },
+        "xdm:additionalIDs": {


Additional Ids are also needed to support multiple ECID --> CRMID mappings that are provided by co-op and private graph customers. The support for additional Identities should be done outside of the Identity object where each id specified, is of type xdm:Identity. Either in the EndUserIds structure or in a separate Identity extension for ExperiencEvent.

@harleensahni please review with Rahul, I agree on your proposal, this is likely just something you need to talk through.

@biswas-adobe has concerns with Identity itself being modified. It's beginning to look like a better approach to using EndUserID here (as well as in Experience Event) is to go with an array of Identities.

Modeling with additional IDs is not compatible with #459 since it does not allow for different authentication states for each id in a namespace.

@harleensahni I also feel like if would be better to use an array of Identity. This is how the same problem was handled in Profile.

@kstreeter, I can change to array of Identity. The problem is that we will face this same issue on EndUserIDs itself in other use cases though, and I think we're just punting the issue. There are cases where a customer can send multiple ids for the same Identity namespace on the same call. These will need to be recorded on the ExperienceEvent and that uses EndUserID.

It's also weird that we would model Identities in two different ways for anyone who is consuming these events.

@kstreeter @biswas-adobe I've changed this to be an array of Identity

harleensahni · 2018-08-02T20:15:45Z

schemas/context/experienceevent.example.1.json

+        }
+      },
+      "xdm:profileStitchID": {
+        "xdm:id": "https://data.adobe.io/entities/profileStitchIdentity/1",


need to change this back to @id

mohrahit · 2018-08-06T18:31:03Z

schemas/context/experienceevent.example.1.json

@@ -114,5 +114,48 @@
  },
  "xdm:marketing": {
    "xdm:trackingCode": "marketingcampaign111"
-  }
+  },
+  "xdm:profileStitching": [


A couple of questions:

What is the use case for adding profileStitching to ExperienceEvent?

Is the user expected to populate this via ETL jobs?

To support profile stitching in realtime for use with realtime segmentation
Someone could populate this with an ETL job if they wanted, but that wasn't the use case I was considering. I think it's open enough to support that.

mohrahit · 2018-08-06T18:32:42Z

schemas/context/experienceevent.example.1.json

+          "xdm:code": "AAM"
+        }
+      },
+      "xdm:profileStitchID": {


Is there a reason to add this to each segment realization and each ExperienceEvent? This information can be looked up on demand from the segment metadata.

Yes, because without it, this event isn't complete. You'd have to reference the segment definition to look up profile stitching that occurred in this event (in ExperienceEvent.profileStitching) to what profiles were used to qualify this segment. That doesn't seem ideal. The alternative to avoid that would be to just embed endUserIds here instead of profileStitchID. That would be much more verbose if multiple segments were realized on this call since the same set of endUserIDs would be repeated in multiple memberships. Also, we may be in a position where we want to record information about profile stitching in ExperienceEvent.profileStitching for other use cases down the road. If that happens, then the same set of information would be in ExperienceEvent.profileStitching.endUserIds and each segmentmembership endUserIds.

@mohrahit, another alternative existed also. That was to have segmentmembership be a property under each profile stitch here in ExperienceEvent. That felt too nested and more importantly, assumed that all segmentation happened with profile stitching.

@mohrahit, are you okay with this approach?

…stead od of with EndUserIDs. Remove concept of additionalIDs

profile stitch

8fd0958

harleensahni mentioned this pull request Jul 24, 2018

Payload value types #411

Merged

harleensahni added 2 commits July 24, 2018 09:00

remove payload reference from this branch

bbc1245

up version in package.json

934910d

cdegroot-adobe suggested changes Jul 24, 2018

View reviewed changes

lrosenthol requested changes Jul 24, 2018

View reviewed changes

cdegroot-adobe reviewed Jul 24, 2018

View reviewed changes

cleanup from code review

22529e2

biswas-adobe suggested changes Aug 2, 2018

View reviewed changes

cdegroot-adobe mentioned this pull request Aug 2, 2018

add propensity to segment membership #452

Closed

harleensahni commented Aug 2, 2018

View reviewed changes

harleensahni added 2 commits August 2, 2018 16:59

clean up from pr

3fe5501

move back to @id where appropriate

7e81df1

mohrahit reviewed Aug 6, 2018

View reviewed changes

Model identities for profile stitching with an array of Identities in…

a45a89a

…stead od of with EndUserIDs. Remove concept of additionalIDs

kstreeter added this to the Stabilizing 1.0 milestone Aug 14, 2018

kstreeter added 3 commits August 14, 2018 16:56

Merge branch 'master' into add-profile-stitch-to-segment-membership

7f90489

Merge branch 'master' into add-profile-stitch-to-segment-membership

f7fe51f

Merge branch 'master' into add-profile-stitch-to-segment-membership

9018d5a

kstreeter merged commit e0e661a into adobe:master Aug 15, 2018

kstreeter mentioned this pull request Aug 16, 2018

Minor refactor of segment membership schemas to be more tool friendly #474

Merged

harleensahni mentioned this pull request Sep 4, 2018

Correct profile stitching #504

Merged

profile stitch #444

profile stitch #444

Conversation

harleensahni commented Jul 24, 2018 • edited by kstreeter Loading

cdegroot-adobe left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cdegroot-adobe commented Jul 24, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

harleensahni commented Jul 26, 2018

lrosenthol commented Jul 26, 2018

harleensahni commented Jul 26, 2018

lrosenthol commented Jul 26, 2018

harleensahni commented Jul 26, 2018

harleensahni commented Jul 26, 2018

lrosenthol commented Jul 26, 2018

cdegroot-adobe commented Jul 26, 2018

harleensahni commented Jul 26, 2018 • edited Loading

lrosenthol commented Jul 26, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

harleensahni commented Jul 24, 2018 •

edited by kstreeter

Loading

harleensahni commented Jul 26, 2018 •

edited

Loading