[Sharethrough] Add CCPA support (prebid#1263)

* Handle gzip responses from ad server correctly

* Bump to version 8

* [Go Modules] Add proxy (prebid#1079)

* Add SSL cert for accessing stored request API (prebid#1087)

* [misspell] fix a misspell (prebid#1102)

* update static bidder params for rubicon video to follow the json marshalling names (prebid#1100)

* Switching yieldmo auction endpoint from http to https (prebid#1103)

* Add Datablocks Adapter (prebid#1095)

* datablocks bid adapter

* ttx

* add test json

* add coverage

* redo ttx

* formatted

* better error handling

* additional tests and recomended fixes

* Adding translatecategories flag to includebrandcategory (prebid#1098)

* Making IAB category translation optional with translatecategories boolean in request

* Updating exchange unit tests to remove extra bids

* Updates from code review comments

* Removed comment about default TranslateCategories value
* Changed translateCat to translateCategories in tests
* Combined helper functions in exchange_test related to TranslateCategories

* Bid floor (prebid#1085)

* Currency handling fix (prebid#1097)

* facebook adapter refactor (prebid#1064)

* Kubient adapter (prebid#1094)

* [synacormedia] Update user sync url to be https (prebid#1115)

This detail was missed while setting up the adapter, but we would like to use https for the user sync.

* Remove Go 1.11 Build Target (prebid#1109)

* Set "Secure" on Same SIte cookies (prebid#1119)

* TripleliftNative Adapter (prebid#1114)

* ignore swp files

* start small

* start really small

* add a user sync

* justify

* triplelift adapter

* add our endpoint

* fix syntax

* config stuff

* compiler fixes

* more config

* add params

* making progress

* make our ext more exty

* start making responses

* more logic

* fix compilation errors

* can we just nil this out?

* augment our json

* radically simplify our json

* fix errs

* infer the bid type

* fix syntax

* fix comilation errors

* rename

* fix compilation error

* config stuff

* simplify params

* more config stuff

* fixes

* revert this

* fix up the extension

* getting closer

* add a test

* update config

* update bidder params

* add the floor here, too

* add a usersync test

* validation, ws, and a test

* update tests

* fix test

* update email

* why not

* change email

* preprocess requests

* do some parsing

* take care of some errors

* floor is optional

* ws

* remove native

* everything is either banner or video

* this should be a float

* floor to floor

* fix compilation errors

* add some tests

* more tests

* more tests

* simplify

* more progress

* format

* ws

* rm

* don't need this

* fix test

* fix test

* don't ignore swap

* change line back

* report an error if there are no valid impressions for triplelift

* check for either a Banner or Video object on the impression

* more tests

* mv

* more tests

* update triplelift end point

* send native

* ws

* start changing tests

* fix more tests

* update config

* add redirect to triplelift usersync

* fix supplier id in triplelift_test

* update tl usersync endpoint and test

* fix tl supplier id in test json

* update usersync test template

* adjust inconsistency with test and sync url

* mv

* update packages

* mv

* mv

* update

* fix compilation errors

* rename

* rename some stuff

* rename

* rename

* fix some compilation errors

* ws

* ws

* add the extra info

* add some extra info

* add some files back

* ws and such

* updates

* ws

* fix compilation error

* mv

* rename

* Revert "rename"

This reverts commit 1b77c72.

* Revert "mv"

This reverts commit 52a134d.

* it builds

* cp a file

* cp another file

* fix a test

* fix test

* add the extra info

* ws

* add some logic

* edit comment

* it compiles

* this is now public

* call this

* add the function

* return nil

* seems to be working

* ws

* seems to be working

* ws

* mv

* starting to work

* ws

* add a new function

* ws

* fix tests

* bug fix

* update some stuff

* revert

* take out prints

* fix up diff

* fix up diff

* update ws

* fix

* ws

* omit the triplelift endppint

* Revert "omit the triplelift endppint"

This reverts commit 7abc3e4.

* populate the endpoint through the extinfo

* ws

* set disabled to be default

* ws

* update types

* fixing tests

* making progres

* fix tests

* fix tests

* more fixes for tests

* fixed tests

* just use a comment

* get rid of endpoint

* restore endpoint

* add some errors around unmarshalling

* ws

* ws

* use the literal

* ws

* ws

* update json

* simplify

* ws

* restore tests

* fail fast when grabbing invcode

* use the right type

* use a different error type

* bump code coverage

* add a new test

* change error type

* ws

* break out test into its own function

* JSON block that has a full data-center specific URL cache info (prebid#1104)

* Update Dockerfile and Makefile  (prebid#1099)

* Add option for running tests as part of the docker image building

* Update Makefile

- Add ability to execute adapter specific tests
- Execute targets for "all" rather than just printing the target name and usage
- Remove use of non-existing "install" target from .PHONY targets
- Remove "build" as a dependency for "image"

* enable app requests for audience network (prebid#1122)

* [docs] fix markdown title (prebid#1124)

* Prometheus Refactor (prebid#1108)

* update default sync url (prebid#1127)

* Update sync url for BidderGrid adapter (prebid#1120)

* [SonarCloud] Legacy auction endpoint (prebid#1017)

* [currency converter] allow to deduce reverse rate (prebid#1126)

This CL allows the currency rate currency to deduce a currency rate even
if not directly defined in the table but the reverse rate is present.

E.q.

USD => EUR is 1.0897
EUR => USD is not set

Old behavior when asking rate from EUR to USD will not be found,
New behavior is using the known reverse rate to deduce the rate.

Rate for 2 USD will be 2 * (1 / 1.0897)

* Updated handleError arguments to be pointers for video endpoint (prebid#1128)

* Updated handleError arguments to be pointers for video endpoint

* Removing unneeded pointer to http.ResponseWriter

* Adding units test for update to handleError

* Revert changes to GetExtCacheData() made in prebid#1104 (prebid#1130) (prebid#1131)

* Better native request validation (prebid#1132)

* require the caller to define native assets[...].ID (prebid#1123)

* require the caller to define native assets[...].ID

* Update assets-with-partial-ids.json

* CCPA Phase 1: AMP Endpoint (prebid#1125)

* facebook: removed Auth-Token from header (replaced by authentication_id in the request body) (prebid#1113)

* Setuid Fix (prebid#1121)

* Update http refresh to use url builder. Fixes prebid#1065 (prebid#1133)

* Add mapping of user.ext.eids[] for LiveIntent in Rubicon bidder (prebid#1089)

* support facebook app_secret config param (prebid#1139)

* CCPA Phase 1: Cookie Sync (prebid#1135)

* null check banner.h (prebid#1142)

* Add Pubnative Adapter (prebid#1134)

* Adding the passing of CCPA value to the bid request for video endpoint (prebid#1143)

* first draft (prebid#1137)

* CCPA Phase 2: Enforcement (prebid#1138)

* Gamoshi Adapter: Update cookie sync (prebid#1146)

* Simplify static/bidder-params/triplelift_native.json (prebid#1152)

* Added US Privacy support in TheMediaGrid server adapter (prebid#1147)

* Add TheMediaGrid server adapter

* Add video support in TheMediaGrid s2s adapter

* Update sync url for TheMediaGrid s2s adapter

* Added CCPA support for TheMediaGrid s2s adapter

* Fix sync url for TheMediaGrid adapter

* CCPA User Sync Updates (prebid#1153)

* Marsmedia - add new bidder (prebid#1118)

* Add Applogy adapter (prebid#1151)

* enforce video.size_id for video imps in rubicon adapter (prebid#1101)

* Updated PubMatic endpoint to use https (prebid#1155)

* Update Example AppNexus Placement ID (prebid#1160)

* Fix Currency Converter Doesn't Output CUR (prebid#1154)

* Add custom JSON req/resp data to the analytics logging… (prebid#1145)

* Add custom JSON req/resp data to the analytics logging for the /openrtb2/video endpoint.

* Add calls in unit tests to cover logging and jsonify of video object.

* CCPA User Sync URL Updates (prebid#1157)

* Fixes audienceNetwork adapter ignoring banner.format sizes. (prebid#1164)

* adding yieldmo vendor id to usersync (prebid#1166)

* Add SmartRTB adapter (prebid#1071)

* Added new adapter for CPMStar ad network banners and video (prebid#1159)

* Update the Conversant sync pixel (prebid#1161)

* Add imp.ext.is_rewarded_inventory flag for rewarded video in Rubicon (prebid#1170)

* [currencies] fix GetInfo() null ref issue (prebid#1169)

This CL fixes the null ref on `RateConverter.GetInfo()` when rates
are nil. Issue: prebid#1136

* Fix triplelift User Sync (prebid#1173)

* Enhance Message For Cache Errors (prebid#1175)

* Fix PubMatic Usersync URL (prebid#1178)

Co-authored-by: pm-isha-bharti <isha.bharti@pubmatic.com>

* [Synacormedia] Add tagId bidder parameter (prebid#1165)

* Remove all non-secure calls from eplanning adapter (prebid#1179)

* Expose Cache HTTP Settings (prebid#1184)

* Adding bid rejection messages to debug response (prebid#1181)

* Adds timeout notifications for Facebook (prebid#1182)

* VIS.X: added app type support (prebid#1194)

* Add Adoppler bidder support. (prebid#1186)

* Add Adoppler bidder support.

* Address code review comments. Use JSON-templates for testing.

* Fix misprint; Add url.PathEscape call for adunit URL parameter.

* Adding support for deal prefixes (prebid#1183)

* updating default hard-coded list of certs (prebid#1201)

Co-authored-by: Shalmali Patil <shalmali.patil@pubmatic.com>

* add admixer adapter (prebid#1195)

* Adding copying of gdpr consent string to openrtb bid request (prebid#1189)

* Adding copying of gdpr consent string to openrtb bid request

* Updated video request to use OpenRTB Video and User objects

* Fixing unit test failure message

* Updates from code review comments

* Updating unit test initialization

* Updated mimes array construction

* fix conversant sync pixel (prebid#1208)

* openx adapter: forward bid response currency in openx adapter if set (prebid#1211)

it was always set to the default USD before

* add ucfunnel adapter (prebid#1192)

* Update required params for TheMediaGrid adapter (prebid#1188)

* add zeroclickfraud adapter (prebid#1207)

* add zeroclickfraud adapter

* fixes for PR

* fix casing of Zeroclickfraud

* Fix Adform's parameters regex (prebid#1214)

* Added adform info file

* Added Adform adapter and bidder

* Updates from master

* Removed usersyncInfo from Adform adapter. Inverted Imp type check.

* Removed excessive loop

* Updated with the last master

* Create readme file for adform

* Fix Adform's parameters regex

Motivation: catastrophic backtracking during regex execution

Details:
- https://regex101.com/r/NNQrWq/1
- string to check "url_domain:keskustelu.suomi24.fi,url_path:/matkailu/matkakohteet/aasia,layout:lg,categories:Matkailu,main_category:Matkailu"

Co-authored-by: v.statkevich <v.statkevich@adform.com>
Co-authored-by: Olga Linkevich <o.linkevich@adform.com>

* If Device.UA is not present in request body, init it with user-agent from header (prebid#1219)

* If Device.UA is not present in request body, init it with user-agent from request header if it's present

* Moved User-Agent handler to parseVideoRequest func and added unit test

* Minor clean up

Co-authored-by: Veronika Solovei <veronika.solovei@xandr.com>

* Queued request timeout (prebid#1217)

Co-authored-by: Veronika Solovei <veronika.solovei@xandr.com>

* docs: adding currency support section (prebid#1199)

* Add ValueImpression Adapter (prebid#1204)

* Kidoz adapter (prebid#1210)

Co-authored-by: Ryan Haksi <ryan.haksi@freshgrade.com>

* Update auction.md (prebid#1224)

Fix type

* Update auction.md (prebid#1225)

Fix typo.

* Added logging to cache for video endpoint (prebid#1220)

* WIP added logging to cache for video endpoint

* Updating cache call to use TTL from config

* Updates from initial feedback
* Log now includes HTTP headers
* Fixed caching to use a new cache entry rather than appending to the
VAST
* Added feature where is query is set, the test flag is set in the
request

* Updated recorded response and handleError

* Updates from code review comments

* Changed recorded output to be only the debug ext
* Removed extra marhal calls
* Changed cache to be an endpoint dependency
* Added debugLog struct to hold all debug related info
* Numerous smaller changes

* Further code cleanup and added unit tests for debug changes

* Added missing error checks

* Added unit test for error case

* added VISX vendor ID for usersyncing (prebid#1229)

Co-authored-by: Aadesh Patel <aadesh.patel@xandr.com>

* First pass at phase 1 TCF 2.0 support (prebid#1228)

* First pass at phase 1 TCF 2.0 support

* minor fixes

* Update go-gdpr library and fix stuff

* Fixes for PR comments

* Updated price granularity unmarshal to accept empty values and ranges (prebid#1230)

* Update vendorID for TheMediaGrid s2s Bid Adapter (prebid#1232)

* treat 204 from FAN as a no bids response (prebid#1233)

Co-authored-by: Aadesh Patel <aadesh.patel@xandr.com>

* AMP CCPA Fix (prebid#1187)

* Update rubicon.md (prebid#1234)

* adding schain interface (prebid#1203)

* added Rewarded Video section (prebid#1200)

also edited all examples so they include the full openRTB context

* nanointeractive adapter (prebid#1213)

* nanointeractive adapter

* nanointeractive adapter, changes after review

* nanointeractive adapter

* nanointeractive adapter, changes after review

* formatting

* Typos Fix (prebid#1236)

* Fix Typo

* Fixed More Typos

* Moved hb_pc_cat_dur modification to be before caching (prebid#1250)

* Handle CCPA + enable gzip response

[#169984259]

* Addressing review (prebid#273)

[#169984259]

* Remove custom gzip logic  (prebid#280)

* Getting rid of custom gzip logic

[#169984259]

* Restore prod ad server url

[#169984259]

Co-authored-by: Benjamin <b.chastanier@criteo.com>
Co-authored-by: guscarreon <guscarreon@gmail.com>
Co-authored-by: Aadesh <aadeshp95@gmail.com>
Co-authored-by: Winston-Yieldmo <46379634+Winston-Yieldmo@users.noreply.github.com>
Co-authored-by: htang555 <htang555@gmail.com>
Co-authored-by: Cameron Rice <37162584+camrice@users.noreply.github.com>
Co-authored-by: ah-tappx <46002207+ah-tappx@users.noreply.github.com>
Co-authored-by: hhhjort <31041505+hhhjort@users.noreply.github.com>
Co-authored-by: Marsel <taipovm@gmail.com>
Co-authored-by: Corey Kress <coreykre@buffalo.edu>
Co-authored-by: Scott Kay <noreply@syntaxnode.com>
Co-authored-by: Kevin Kerr <kkerr@triplelift.com>
Co-authored-by: Mansi Nahar <mansinahar@users.noreply.github.com>
Co-authored-by: Benjamin <benjamin.chastanier@gmail.com>
Co-authored-by: TheMediaGrid <44166371+TheMediaGrid@users.noreply.github.com>
Co-authored-by: Austin Bischoff <Austinb@users.noreply.github.com>
Co-authored-by: rpanchyk <rpanchyk@users.noreply.github.com>
Co-authored-by: Florian Hartwig <florian.hartwig@pubnative.net>
Co-authored-by: Salomon Rada <salomon@gamoshi.com>
Co-authored-by: vladi-mmg <vladi@m-m-g.com>
Co-authored-by: Aleksei Lin <aleksey@incsw.in>
Co-authored-by: PubMatic-OpenWrap <UOEDev@pubmatic.com>
Co-authored-by: jmaynardxandr <46759873+jmaynardxandr@users.noreply.github.com>
Co-authored-by: evanmsmrtb <evanm@smrtb.com>
Co-authored-by: CPMStar <josh@cpmstar.com>
Co-authored-by: johnwier <49074029+johnwier@users.noreply.github.com>
Co-authored-by: pm-isha-bharti <isha.bharti@pubmatic.com>
Co-authored-by: Seba Perez <sperez@est.frba.utn.edu.ar>
Co-authored-by: Michael Kuryshev <mk0x9@users.noreply.github.com>
Co-authored-by: Viacheslav Chimishuk <vchimishuk@yandex.ru>
Co-authored-by: Shalmali Patil <shalmali.patil@pubmatic.com>
Co-authored-by: DmitryStashkevich <34479135+DmitryStashkevich@users.noreply.github.com>
Co-authored-by: vstatkevich <v.statkevich@gmail.com>
Co-authored-by: v.statkevich <v.statkevich@adform.com>
Co-authored-by: Olga Linkevich <o.linkevich@adform.com>
Co-authored-by: Veronika Solovei <kalypsonika@gmail.com>
Co-authored-by: Veronika Solovei <veronika.solovei@xandr.com>
Co-authored-by: bretg <bgorsline@gmail.com>
Co-authored-by: thuyhq <61451682+thuyhq@users.noreply.github.com>
Co-authored-by: rhaksi-kidoz <61601767+rhaksi-kidoz@users.noreply.github.com>
Co-authored-by: Ryan Haksi <ryan.haksi@freshgrade.com>
Co-authored-by: ACannuniRP <57228257+ACannuniRP@users.noreply.github.com>
Co-authored-by: Aadesh Patel <aadesh.patel@xandr.com>
Co-authored-by: Rade Popovic <32302052+nanointeractive@users.noreply.github.com>

adapters/sharethrough/butler.go

@@ -7,6 +7,7 @@ import (
 	"github.com/prebid/prebid-server/adapters"
 	"github.com/prebid/prebid-server/errortypes"
 	"github.com/prebid/prebid-server/openrtb_ext"
+	"github.com/prebid/prebid-server/privacy/ccpa"
 	"net/http"
 	"net/url"
 	"regexp"
@@ -21,6 +22,7 @@ type StrAdSeverParams struct {
 	BidID              string
 	ConsentRequired    bool
 	ConsentString      string
+	USPrivacySignal    string
 	InstantPlayCapable bool
 	Iframe             bool
 	Height             uint64
@@ -94,13 +96,19 @@ func (s StrOpenRTBTranslator) requestFromOpenRTB(imp openrtb.Imp, request *openr
 		return nil, err
 	}
 
+	usPolicySignal := ""
+	if usPolicy, err := ccpa.ReadPolicy(request); err == nil {
+		usPolicySignal = usPolicy.Value
+	}
+
 	return &adapters.RequestData{
 		Method: "POST",
 		Uri: s.UriHelper.buildUri(StrAdSeverParams{
 			Pkey:               pKey,
 			BidID:              imp.ID,
 			ConsentRequired:    s.Util.gdprApplies(request),
 			ConsentString:      userInfo.Consent,
+			USPrivacySignal:    usPolicySignal,
 			Iframe:             strImpParams.Iframe,
 			Height:             height,
 			Width:              width,
@@ -184,6 +192,9 @@ func (h StrUriHelper) buildUri(params StrAdSeverParams) string {
 	v.Set("bidId", params.BidID)
 	v.Set("consent_required", fmt.Sprintf("%t", params.ConsentRequired))
 	v.Set("consent_string", params.ConsentString)
+	if params.USPrivacySignal != "" {
+		v.Set("us_privacy", params.USPrivacySignal)
+	}
 	if params.TheTradeDeskUserId != "" {
 		v.Set("ttduid", params.TheTradeDeskUserId)
 	}

adapters/sharethrough/butler_test.go

@@ -437,6 +437,7 @@ func TestBuildUri(t *testing.T) {
 				BidID:              "bid",
 				ConsentRequired:    true,
 				ConsentString:      "consent",
+				USPrivacySignal:    "ccpa",
 				InstantPlayCapable: true,
 				Iframe:             false,
 				Height:             20,
@@ -450,6 +451,7 @@ func TestBuildUri(t *testing.T) {
 				"bidId=bid",
 				"consent_required=true",
 				"consent_string=consent",
+				"us_privacy=ccpa",
 				"instant_play_capable=true",
 				"stayInIframe=false",
 				"height=20",

adapters/sharethrough/sharethrough.go

@@ -10,7 +10,7 @@ import (
 )
 
 const supplyId = "FGMrCMMc"
-const strVersion = 7
+const strVersion = 8
 
 func NewSharethroughBidder(endpoint string) *SharethroughAdapter {
 	return &SharethroughAdapter{