First Party Data Revision

[bretg]

Type of issue

Improvement

Description

The Taxonomy Taskforce has proposed a revision of Prebid's First Party Data (FPD) conventions for the following reasons:

1. The mappings between PBJS conventions and OpenRTB are harder to follow than necessary.
2. There's a mismatch between PBJS and PBS:
   1. The PBJS First Party Data spec says that publishers should put standard OpenRTB attributes in context and arbitrary FPD into into context.data
   2. This causes data to be sent to Prebid Server in site and site.data.
   3. However, Prebid Server is expecting to receive arbitrary values at site.**ext**.data
3. Context -vs- Content is confusing.
4. The original documentation showed user.data as an object when it needs to be an array.

Since there are only ~10 bidders currently using the FPD conventions, we propose migrating FPD to a pure-OpenRTB format.

Proposal

1. Going forward, we recommend everything under setConfig({ortb2: OBJECT}) just becomes the OpenRTB representation. This allows the pbsBidAdapter and supporting bidders to simply do a JSON merge of the ortb2 object into the final request. It also allows us to get rid of one-off docs like site.content.language and app support.

• site object
  • site.ATTR - standard OpenRTB attributes (e.g. keywords)
  • site.ext.data -- arbitrary publisher attributes (e.g. editor_rating=4)
  • site.content.data[] -- content 'segments'
• app object (if post-bid)
• user object
  • user.ATTR - standard OpenRTB attributes (e.g. gender)
  • user.ext.data -- arbitrary publisher attributes (e.g. job_cat=sales)
  • user.data[] - user 'segments'

2. Ad-Unit Specific FPD is likewise placed under 'ortb2Imp' in the AdUnit. It will get merged into imp[]. We recommend moving "context.data" to "ext.data".

3. We create a module to validate FPD:

   • verify OpenRTB datatypes, remove/warn any that are likely to choke downstream readers
   • verify that certain OpenRTB attributes are not specified: just imp for now
   • optionally suppress user FPD based on a TBD opt-out signal
   • populate available data into object: referer, meta-keywords, cur

4. Data permissioning stays the same: use of thesetBidderConfig() function.

Work Items

1. The setConfig and setBidderConfig functions support a transition period where they map the original 'fpd' config:
   • fpd.context.ATTR --> ortb2.site.ATTR
   • fpd.context.data.ATTR --> ortb2.site.ext.data
   • fpd.user.ATTR --> ortb2.user.ATTR
   • fpd.user.data.ATTR --> ortb2.user.ext.data. (for fpd only... ortb2 can specify user.data)
2. In order to maintain backwards compatibility for publishers using AdUnit.fpd, at the start of the auction, PBJS-Core should move AdUnit.fpd.context.data to ortb2Imp.ext.data and AdUnit.fpd.context.pbAdSlot to ortb2Imp.ext.data.pbadslot. It can be assumed that ortb2Imp will not already be in the adunit -- the publisher cannot have BOTH fpd and ortb2Imp defined at the same time.
3. gptPreAuction:
   a) move adunit.fpd to adunit.ortb2
   b) adUnit.ortb2Imp.ext.data.adserver.{name, adslot}
   b) pbAdSlot moves to AdUnit.ortb2Imp.ext.data.pbadslot
4. Two bid adapters (Rubicon and LuponMedia) updated to look for pbadslot also in ortb2Imp.ext.data.pbadslot
5. pbsBidAdapter
   a) merge the new ortb2 and AdUnit.ortb2Imp.ext objects into the OpenRTB JSON.
   b) therefore imp[].ext.context.data.pbadslot is now changed to imp[].ext.data.pbadslot (no context)
   c) read adUnit.ortb2Imp.ext.data.adserver from the new location. Output location is moved to imp[].ext.data.adserver (no context)
6. PBJS bidAdapters and RTD modules that support fpd will be converted to ortb2: adrelevantis, amx, avocet, criteo, grid, haloRtd, imar, jwPlayerRtd, luponmedia, rubicon, smaato, triplelift
7. SDK: no changes for now. Support for data segments will be a future project.
8. PBS-Java:
   a) merge imp[].ext.context.data.ATTR to imp[].ext.data.ATTR
   b) Confirm that {site,app}.content.data[] and user.data[] are passed to adapters
   c) Update PG targeting so that pbAdSlot can come in imp[].ext.data.pbadslot (no context)
   d) Update PG targeting so adserver can come in imp[].ext.data.adserver (no context)
   e) if user.data is an array, keep it as-is. If user.data is an object, to merge into user.ext.data and remove user.data.
9. A new optional "first party data" module pulls in page context and does validation. See First Party Data Module #6099
10. Documentation updated

Examples

Example Global FPD

pbjs.setConfig({
   ortb2: {
       site: {
           name: "example",
           domain: "page.example.com",
           cat: ["IAB2"],
           sectioncat: ["IAB2-2"],
           pagecat: ["IAB2-2"],
           page: "https://page.example.com/here.html",
           ref: "https://ref.example.com",
           keywords: "power tools, drills",
           search: "drill",
           content: {
              data: [
                  segment: [ ... ]
              ]
           }
           ext: {
               data: {   // fields that aren't part of openrtb 2.5
                   pageType: "article",
                   category: "repair"
                }
           }
        },
        user: {
           yob: 1985,
           gender: "m",
           keywords: "a,b",
           data: [
               segment: [ ... ]
           ],
           ext: {
              data: {
                  registered: true,
                  interests: ["cars"]
              }
           }
        }
    }
});

Example AdUnit-specific FPD

pbjs.addAdUnits({
    code: "test-div",
    mediaTypes: {
        banner: {
            sizes: [[300,250]]
        }
    },
    ortb2Imp: {
         ext: {
            data: {
                pbadslot: "homepage-top-rect",
                adUnitSpecificAttribute: "123"
            }
         }
    },
    ...
});

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[GLStephen]

What about ortb2 being an array call ortb, and each "ortbData" object having a .version property string. Then you could pass 2.x and 3.x and 4.x data through it? I know it sounds annoyingly complicated, but it matches some of the approach already in ORTB spec.

[bretg]

@patmmccann and I discussed this.

The goal is that the value of the object is pure OpenRTB. It doesn't make sense to put the version inside the object because then code wouldn't know what version it without parsing. Chicken-and-egg problem.

Another approach is to have the version as a peer of the 'ortb' object. e.g. 'ortbVersion'. Then code that supports multiple versions would have to look at both values to figure out what to do.

We ended up deciding the external version info didn't simplify, usage, reading, or transition. In fact, the hypothetical transition to ortb3 actually seemed a little better with the version right in the attribute name because then we have the option of allowing the publisher or a translation module to specify BOTH ortb2 and ortb3 concurrently.

[bretg]

I updated the AdUnit-specific part of this from ortb2ImpExt to ortb2Imp. This will enable future use cases where the publisher wants to pass imp-level data and not just imp.ext data.

[bretg]

@mmoschovas has taken an initial cut at FPD 2.0. He found that several bid adapters are passing the FPD directly to their endpoints and parsing it there. Which means this could be a breaking change for these adapters, so we propose adding a reverse mapping directly in your adapters until you confirm your endpoints can handle the updated structure, at which time you can remove the conversion code from the adapter.

Here is the list of bidders that copy out fpd from the config and pass it along as is.

• admixer @AdmixerTech
• amx @nickjacob
• avocet @scritchley
• inmar @zandree-owneriq

So for example, here's how admixer passes it:

    const payload = {
      imps: [],
      fpd: config.getConfig('fpd')
    };

The proposal is to create a backwards-compatible global FPD data-structure in each by reading ortb2 config and:

• copy site.ext.data.ATTR to fpd.context.data.ATTR
• copy site.ATTR (not ext) to fpd.context.ATTR
• copy user.ext.data.ATTR to fpd.user.data.ATTR

If your adapter reads AdUnit-specific FPD, we'll create a backwards-compatible structure by reading ortb2Imp.ext.data and copying everything to fpd.context.data.

In case you're reading pbAdSlot, which has moved, we'll copy ext.data.pbadslot to fpd.context.pbAdSlot

Here is the list of bidders that read fpd.context and fpd.user separately from the config and pass them along in two pieces:

• criteo @leonardlabat , @allanjun
• grid @TheMediaGrid
• luponmedia @adxpremium
• smaato @sbrosinski
• triplelift @kzhouTL , @colbertk , @davidwoodsandersen

So for example:

    Object.assign(bidderRequest, {
      publisherExt: config.getConfig('fpd.context'),
      userExt: config.getConfig('fpd.user'),
      ...
    });

    const configKeywords = utils.transformBidderParamKeywords({
      'user': utils.deepAccess(config.getConfig('fpd.user'), 'keywords') || null,
      'context': utils.deepAccess(config.getConfig('fpd.context'), 'keywords') || null
    });

We're proposing to do a similar compatibility conversion for these adapters -- once your endpoints support the updated location, you can remove the conversion code from your endpoints.

We're not quite sure how to handle Audigent's HaloRTD module -- looks to us like it expects AdUnit.bid.fpd.user.data, which isn't a scenario we specifically supported. (We actually ruled out adunit-specific user FPD, but perhaps you instruct your customers to add it?) Adding @anthonylauzon.

[antlauzon]

Can you provide more information about ad-unit user fpd issue? How is a bidder supposed to consume the site, user and ad-unit fpd segments and then combine them server-side? Wouldn't it make more sense to provide user fpd on the ad unit level if that is all that bidder is consuming? It sounds like what we actually want to do within the scope of this protocol is simply add fpd user data and avoid decorating ad units. Does that make sense? If so, I will update our module accordingly.

[bretg]

How is a bidder supposed to consume the site, user and ad-unit fpd segments and then combine them server-side?

@anthonylauzon - there isn't a right way. To be honest, the first design of this didn't make it easy since it wasn't based on OpenRTB. Which is why we're trying to align it.

Wouldn't it make more sense to provide user fpd on the ad unit level if that is all that bidder is consuming?

Basically, the FPD page will be updated to something like this:

If your protocol is OpenRTB-based, all you have to do to get the global FPD is merge in the ortb2 config:

config.getConfig('ortb2');

Then to pull in the adunit-specific FPD, you would merge AdUnit.ortb2Imp into the specific imp[] object you're making for that adunit.

[antlauzon]

Does this allow for dynamic ortb user segment data at the user level? How can I add user segment data to the fpd user's "segment" field without calling setConfig inside of the RTD module?

[bretg]

@anthonylauzon - there's no way to set ortb2 without setConfig.

See #6425 for a proposal -- mergeConfig.

Closing this issue as FPD2 is live. Refinements to the RTD module interface can take place 6425 or other issues.