BigQuery Release Date

General
1

We’re using the OpenTargets as a public dataset available in BigQuery . The timestamp shows the last modification as Nov 2023. Is this dataset always the most up-to-date version ? Asking since the Release Notes for 24.03 mentions March 2024 …

2

Hi @Karan_Randhawa, and welcome to the Open Targets Community! :tada:

The Open Targets platform public dataset on Google BigQuery is updated with the 24.03 (March 2024 release). The sync happens on a monthly basis.

You can confirm the release version by querying the ot_release table in the open_targets_platform dataset.

image
image2891×1377 253 KB

The dataset in BigQuery is just a placeholder for all the underlying tables. The timestamp on this dataset only gets updated if the metadata is updated which might not be with every release.
However the data and any underlying tables are updated with every release.

Do let me know if this answers your query.

Best,
Prashant Uniyal

1 Like
3

Thanks Prashant . Would you know if there’s a way to create a synced copy of a public dataset in our own BigQuery production instance ? I’m wondering what the implications for query performance, costs might be if we query this dataset directly.

4

Hi Karan,

I think you can use the BigQuery copy function or the BigQuery Data Transfer Service to schedule recurring copies of the data in the Google Cloud console or use the following command on bq CLI:

bq mk \
  --transfer_config \
  --project_id=PROJECT \
  --data_source=cross_region_copy \
  --target_dataset=DATASET \
  --display_name=NAME \
  --service_account_name=SERCICE_ACCOUNT \
  --params='PARAMETERS'

You can find more details here and here.

I assume there would not be any charges for exporting data from BigQuery to Google Cloud storage bucket, but you would incur charges for storing the exported data in Cloud Storage. The egress charges would also apply whenever you export the data from the cloud bucket to your local setup outside the Google Cloud infrastructure.

The cost of accessing and storing your own copy of the public data would be charged according to the Google’s BigQuery pricing structure.

Hope this helps.

Regards,
Prashant Uniyal

1 Like