Home

Helpful Recipes

Bulk-delete BigQuery Datasets

To delete a bunch of bigquery datasets, do this at a bash prompt, replacing temp_dataset_ with whatever match string you want.

for i in $(bq ls -n 9999  | grep temp_dataset_ | awk '{print $1}'); do echo bq rm -rf  $i; done;

Then after you are happy with the output, delete echo and run it for real

Bulk-delete BigQuery Tables in parallel

bq ls -n 9999 PROJECT:DATASET | grep TABLE_PATTERN | awk '{print $1}' | xargs -P 8 -I{} bq rm -f DATASET.{}

-P 8 runs 8 processes in parallel

get the schema from an existing Bigquery table

bq show --format=prettyjson bigquery-public-data:san_francisco.bikeshare_trips | jq '.schema'

Export data from a query

• First, run the query and save the result to a bq table
• Then export to GCS as newline json
• Then download the result