Provide ability to set more options when outputting joined data to bq table.

[mavysavydav]

Is your feature request related to a problem? Please describe.

We need to be able to set expiries for the tables, project.dataset.tableName, and also option for create vs replace.

Describe the solution you'd like

Ideally the user can optionally pass in a config object like QueryJobConfig. QueryJobConfig has many options such as destination which allows one to set the destination of table and name, but it doesn't seem like it can handle create/replace (it has create_disposition which offers some limited control) and expiry.

Describe alternatives you've considered

N/A. Willem and a few of us briefly discussed this and he played around with the idea of passing in and out sql queries. So one could modify the query and pass it back in.

[woop]

@mavysavydav the problem statement makes sense. get_historical_features() is a general (non-implementation specific) method, but the retrieval jobs are specific to the implementation. So roughly there are options like

job = get_historical_features(..., kwargs)

where those options get passed all the way to the offline store and the offline store then knows how to use them

job = get_historical_features(...)
job.set_config(QueryJobConfig)
job.to_bigquery()

job = get_historical_features(...)
job.to_bigquery(..., config=QueryJobConfig)

job = get_historical_features(...)
sql = job.to_sql()
job_config = bigquery.QueryJobConfig(default_dataset="bigquery-public-data.stackoverflow")
client.query(sql, job_config=job_config)

My preference is (3) or (4) right now, but don't feel super strongly.

[codyjlin]

🙋 working on this now! Please assign me.

[mavysavydav]

yea agreed 3 or 4 is good. cody will look into

[mavysavydav]

solution 4 would work well with in-house wrappers. The wrapper could still present a clean api for users even tho it's changing the sql to add things like expiry. Though expiry seems like something feast would want to support natively. Tho maybe we'll wait for more needs to emerge b4 natively supporting it.