Prebid Server OpenRTB 2.6 support

[bretg]

OpenRTB 2.6 is open for comment at https://iabtechlab.com/wp-content/uploads/2022/04/OpenRTB-2-6_FINAL.pdf

A field-by-field list of changes below, but here are the high-level changes for Prebid Server:

1. Logically, stored requests need to be merged before doing the ORTB normalization. Or it needs to be done twice. We shouldn't assume that PBS-host companies have converted all of their stored requests at initial release.
2. Expand JSON validation. We propose keeping the removed fields so PBS validation is done on a superset of 2.5 and 2.6.
3. Transition support for moved objects: schain, eids, rewarded video. Clients will be placing this data on the original extensions for years to come. We could make this transition support modular or configurable so host companies can avoid the cost at some point.
   a. The committee has agreed to move extended locations to the new locations. e.g. source.ext.schain to source.schain.
   b. If both new and old locations are seen in a request (e.g. post SRID merge), prefer the new location.
4. Updates to the ortb2 blocking module to support cattax fields
5. Port long-form video functionality to the /ortb2/auction endpoint using the new pod-related fields.
6. If a bidder only supports 2.5:
   1. PBS-core should copy these objects from the 2.6 location to the 2.5 location: schain, eids, regs, user.consent, rewarded video.
   2. For fields that don't exist in 2.5 (e.g. imp.ssai), PBS-core may either remove them or leave them in place. Theoretically bid adapters should ignore them.
7. Bidder-specific first party data must support specifying all the new fields.

These changes don't all have to come at once.

BidRequest changes in 2.6

• New field $.wlangb. pass it through to bid adapters
• New fields \$.{site,app}.content.langb, \$.device.langb. Pass through to bid adapters. ORTB 2.6 says "Only one of language or langb should be present." but PBS does not need to enforce this.
• New field $.cattax: clarifies .bcat. Will need to update the ortb2 blocking module to support this field
• New fields $.site.cattax. Support as First Party Data fields and pass through to bidders.
• New field $.app.cattax. Support as First Party Data fields and pass through to bidders.
• New field $.{}.publisher.cattax. Support as First Party Data fields and pass through to bidders.
• New field $.{}.content.cattax. Support as First Party Data fields and pass through to bidders.
• New field $.{}.producer.cattax. Support as First Party Data fields and pass through to bidders.
• New fields: $.imp[].{video,audio}.{rqddurs, maxseq, poddur, podid, podseq, mincpmpersec, slotinpod}. Pass through to bidders. Port long form video functionality.
• New field: $.imp[].rwdd. This replaces the extension imp.ext.prebid.is_rewarded_inventory. PBS-core and/or several bid adapters will need to be updated to support a transition period where either value could be present. Adapters looking for the extension flag: algorix, bidmachine, pangle, openx, rubicon, unicorn.
• New field $.imp[].ssai: pass through to bidders
• New objects $.{app,site}.content.{network, channel} pass through to bidders. Note that channel here is not related to ext.prebid.channel.
• New object $.source.schain. Contains the new ortb2 objects SupplyChain, SupplyChainNode. This data currently comes in on .source.ext.schain. It appears to be a lift-and-shift.
  • $.source.schain.nodes.hp is an optional field in ORTB 2.6 where it was required in 2.5.
• New field: $.regs.gdpr. This data currently comes in on .regs.ext.gdpr.
• New field: $.regs.us_privacy. This data currently comes in on .regs.ext.us_privacy.
• New field: $.user.consent. This data currently comes in on .user.ext.consent.
• New fields: $.{app,content,site,user}.kwarray. Pass it through to bid adapters. This is an array version of keywords. ORTB2.6 says "Only one of ‘keywords’ or ‘kwarray’ may be present.", but PBS does not need to enforce this.
• New object: $.user.eids. Contains the new ortb2 objects EIDs and UIDs. This data currently comes in on .user.ext.eids. It appears to be a lift-and-shift.
• New object $.device.sua. Contains new ortb2 object UserAgent.
• Removed fields: $.banner.{wmax, hmax, wmin, hmin}. Propose retaining these fields as deprecated.
• Removed field: $.video.protocol. Propose retaining this field as deprecated.
• Removed field: $.content.videoquality. Propose retaining this field as deprecated.

BidResponse changes in 2.6

• $.seatbid[].bid[].langb,dur, slotinpod, apis : pass them through to response
• $.seatbid[].bid[].cattax: clarifies $seatbid[].bid[].cat. Will need to update the ortb2 blocking module to support enforcement on this field. Update validations in long form video category mapping.
• $.seatbid[].bid[].mtype : this should replace $seatbid[].bid[].ext.prebid.type. Certain clients may depend on the prebid extension location, so PBS-core should look for bidder-supplied mtype and copy to ext.prebid.type for a transition period.

[SyntaxNode]

Thank you for kicking this off @bretg.

I propose keeping the removed fields so PBS validation is done on a superset of 2.5 and 2.6.

I am interested in what the rest of the committee and the community feels about this. I prefer to choose a standard and stick with it rather than creating our own hybrid model. The removed fields in 2.6 (in 2022) were marked as deprecated in 2.5 (2016). and I can find no usage of them in any adapter. This doesn't mean an adapter's bidding server doesn't use them, but I think 6 years is long enough of a deprecated period and am supportive of removing them in PBS. This would allow us to use the true OpenRTB 2.6 model throughout the app and put us in a cleaner position for a future 2.7 version.

Transition support for moved objects: schain, eids, rewarded video.

It's the intention of the IAB for the ecosystem to support both the new promoted locations alongside the existing extension locations, where the newer promoted locations are preferred. This brings up the question of how PBS should behave if both the new and old locations are present? A good topic for discussion.

We could make this transition support modular or configurable so host companies can avoid the cost at some point.

I consider this a decision required for each adapter, rather than for the host company. If an adapter's server supports only 2.5 and the bid request is using the newer 2.6 fields, I propose PBS moves the 2.6 fields back to extensions before invoking the adapter and possibly remove the newer 2.6 fields if they're populated. This would be powered by a per-adapter config option which over time we can encourage adapters to add 2.6 support.

PBS-core and/or several bid adapters will need to be updated to support a transition period where either value could be present. Adapters looking for the extension flag: algorix, bidmachine, pangle, openx, rubicon, unicorn.

I propose we keep the transition period indefinite while PBS remains on the 2.x data model unless the IAB provides guidance to the contrary. We absolutely need to upgrade the core and the adapters listed (thank you for compiling the list). Perhaps we can make helper methods to locate those fields for adapters (and the core) to use.

Port long-form video functionality to the /ortb2/auction endpoint using the new pod-related fields.

Completely agree, which should be no surprise as I've brought this up for discussion in the committee before :). Can we track this in a separate story? This seems like a large enough effort to track and design independently. I've ensured the new 2.6 video fields are compatible with our custom PBS pod model, but I'd also like this opportunity to go back and re-evaluate our previous design decisions. I expect this would ultimately lead to the deprecation of the /video endpoint.

$seatbid[].bid[].mtype

I propose we add validation to this field and reject bids with unknown mtypes. For PBS-Go, adapters should not have to provide the BidType if their bidding server populates this field.

I would also like to discuss how we approach adapters which specify the OpenRTB version as 2.5 in the header to their bidding servers - there are a lot (aceex, acuityads, adkernel, adkernelAdn, adoppler, adpone, advangelists, adyoulike, algorix, apacdex, facebook, bidmachine, bidmyadz, bizzclick, brightroll, coinzilla, gamma, impactify, kidoz, krushmedia, lockerdome, lunamedia, madvertise, marsmedia, nanointeractive, nextmillenium, ninthdecimal, richaudience, silvermob, smartrtb, smartyads, smilewanted, somoaudience, telaria, unicorn, vidoomy, yahoossp). Perhaps this can go hand-in-hand with per-adapter config options for how to handle the transition.

[patmmccann]

I am mostly worried about these:

New object: $.source.schain. Contains the new ortb2 objects SupplyChain, SupplyChainNode. This data currently comes in on $.source.ext.schain. It appears to be a lift-and-shift.

New object: $.user.eids. Contains the new ortb2 objects EIDs and UIDs. This data currently comes in on $.user.ext.eids. It appears to be a lift-and-shift.

Moving the location sounds breaking, copying it sounds better. However, perhaps adapters could somehow say where they want it?

[patmmccann]

re @SyntaxNode 's point on if removed fields are being used; we see very different bidder behavior when we pass different values on protocols field for video. I would be quite nervous to deprecate it.

[SyntaxNode]

perhaps adapters could somehow say where they want it?

Agree.

FWIW: I conveyed to the IAB the problem the ecosystem would encounter if the group decided to promote the extensions, but most members were adamant in providing extensions a path for inclusion into the next spec iteration. I can understand both views. It's less ideal for Prebid, putting more work on our systems to make adapters happy.

[bretg]

@patmmccann , note about your statement

... when we pass different values on protocols field for video ...

Note that the protocols field (the array version) stays. It's the singular protocol field that's gone.

[bretg]

Discussed in committee. Notes:

• 2.6 is still not finalized, we have until summer.
• We could implement bidder-level config declaring whether they support 2.5 or 2.6. The default will start at 2.5.
• Either way, PBS-core will normalize incoming requests to 2.6 by default.
• But if a bidder declares 2.5, PBS-core will normalize back to the older syntax and remove any 2.6 fields.
• We should think of this as two transition periods: incoming requests and adapter support.
  • For incoming requests, we should have metrics for when PBS-core sees 2.5 fields incoming so we can determine whether that normalization is still needed.
  • On the adapter side, we can decide at some point to require bidders to upgrade to 2.6 or be left behind in a major release
• Prebid Server will need to support 2.6 early on so Prebid.js and SDK can adopt it.
• At some point, PBS will be upgraded to support 2.6 only. Adapters not supporting 2.6 will be excluded from the release.
• Adapters could downgrade to 2.5 on their side and declare that PBS-Core doesn't need to do the downgrade for them.

Will continue to discuss in future meetings.

[SyntaxNode]

FYI: $.imp[].rewarded was actually written in the spec as "reward" and will be changing to "rwdd".

[bretg]

rwdd? ugh. updated the description

[SyntaxNode]

rwdd? ugh. updated the description

Really. Designed by committee. :-/

[bretg]

Discussed in committee. The goal is to have PBS-Go and PBS-Java releases in June to support the transition approach.

[bretg]

The IAB has blessed 2.6. Went back through the doc and reviewed. rwdd is still there. Also added device.sua.

[bretg]

Discussed in committee.

• we agreed to move extended locations to the new locations. e.g. source.ext.schain to source.schain.
• if the request specifies the extension on both locations, prefer the new (2.6) location.

[bretg]

Discussed in the video committee. We agreed it would be fine for the PBS 2.6 effort to be delivered in two phases:

• basic 2.6 support
• build out long-form support within the /auction endpoint and deprecate the separate long-form endpoint

[bretg]

Added

• New field: $.regs.gdpr. This data currently comes in on $.regs.ext.gdpr.
• New field: $.regs.us_privacy. This data currently comes in on $.regs.ext.us_privacy.
• New field: $.user.consent. This data currently comes in on $.user.ext.consent.

[bretg]

Added

• New fields $.{content,device}.langb. Pass it through to bid adapters. ORTB 2.6 says "Only one of language or langb should be present." but PBS does not need to enforce this.
• New fields: $.{app,content,site,user}.kwarray. Pass it through to bid adapters. This is an array version of keywords. ORTB2.6 says "Only one of ‘keywords’ or ‘kwarray’ may be present.", but PBS does not need to enforce this.
• $.seatbid[].bid[].tactic: pass through to response

[bretg]

The team has noted "none of listed removed fields has been in PBJ since initial commit." If that's the case, I don't see a need to add old fields as part of this effort.

[bretg]

More clarifications coming from implementation:

1. merge stored requests first, then normalize to ORTB 2.6
2. If both new and old locations are seen in a request, prefer the new location

[SyntaxNode]

• $.seatbid[].bid[].tactic: pass through to response

The tactic id is not new in OpenRTB 2.6. It was introduced in 2.5.

[bretg]

Removed 'tactic' from the list. Also added this item as discussed in committee

7. Bidder-specific first party data must support specifying all the new fields.

We ought to discuss this. My preference is that the bidder-specific FPD code just handles 'All The Fields' so that I don't have to update the PBS FPD doc with an enumerated list of all fields. But if you all tell me there's no way around this, will submit.

[bretg]

Discussed again in committee, specifically the topic of new ORTB fields. We agreed to communication to PBS adapters about new 2.6 fields and note that they should remove fields in their adapter if it's a problem for them

[bretg]

Discussed ORTB 2.6 again in committee today

What does ORTB version support really mean with the more frequent updates, e.g. ORTB 2.6-YYYYMMDD?

• The PBS-Core JSON schema can evolve to cover new fields as needed
• Bid adapters that claim "2.6" are expected to be able to handle new fields that are currently unknown to them. The issue is with fields that get "promoted" from extensions.
• For example, say regs.ext.blah gets popular, then ORTB promotes it to regs.blah.
• At the PBS-core level, we would add regs.blah to the 2.6 schema
• Core would also 'upgrade' regs.ext.blah up to regs.blah
• Would we down-convert for adapters still claiming 2.5?
  • Option 1: support 2.6-YYYYMMDD versioning. If we did this, PBS-core would downgrade any promoted ext fields back to ext until endpoint/adapter ready to receive on the promoted location
  • Option 2: adapters must be immediately upgraded to be able to handle the new location
  • Option 3: when PBS implements a promotion upgrade, it could leave a copy in the old location for a period of time. It wouldn't downgrade if the incoming signal is on the promoted location.
  • Discussed starting simple with Option 3, then add 2.6-YYYYMMDD versioning support as needed in the future.
• ORTB response fields
  • PBS could look at PBJS/SDK version to know where to place response fields.
  • We agreed to cross this bridge when we get to it.

[Pubmatic-Dhruv-Sonone]

@SyntaxNode @bretg
Do we have any prd of ORTB 2.6 support for prebid-server? Also What are the timelines for ORTB 2.6 support for the "/openrtb2/auction" endpoint?

[bretg]

@Pubmatic-Dhruv-Sonone - this issue serves as the requirements. What aspect do you think may be missing? If anything, I can see a benefit in adding a reference document on docs.prebid.org summarizing what fields are supported, downgraded, etc.

What are the timelines for ORTB 2.6 support for the "/openrtb2/auction" endpoint?

Do you mean video podding? /auction is the main endpoint where all 2.6 support has been created.

No committed timeline on getting rid of /video and moving podding into /auction

[Pubmatic-Dhruv-Sonone]

@Pubmatic-Dhruv-Sonone - this issue serves as the requirements. What aspect do you think may be missing? If anything, I can see a benefit in adding a reference document on docs.prebid.org summarizing what fields are supported, downgraded, etc.

What are the timelines for ORTB 2.6 support for the "/openrtb2/auction" endpoint?

Do you mean video podding? /auction is the main endpoint where all 2.6 support has been created.

No committed timeline on getting rid of /video and moving podding into /auction

@bretg I was asking about changes related to using new ORTB 2.6 fields in core auction logic and conversion of bidder request (2.5 to 2.6 and vice versa) based on bidder level config (ortb version).

[bretg]

The auction endpoint has supported some 2.6 fields for many months now.

It's rather confusing because there's no such thing as "2.6". We now live in a world where the IAB adds fields every couple of months. I'll draft a docs page that describes which fields are supported by Go/Java and the "downgrade" behavior.

[bretg]

Note that #3613 has changed the downgrade-to-2.5 procedure.

[SyntaxNode]

Full upgrade/downgrade feature set implemented in PBS-Go v3.0.0.