Hi @marcustutert. Indeed, when you download data from the Google Cloud platform, it applies egress charges. The exact amount depends on your location, the complete table can be found here: All networking pricing | Virtual Private Cloud | Google Cloud.
The egress charges don’t apply when you move the data within the cloud; for example, if you export the BigQuery dataset to a cloud bucket using the command I described below).
However, the egress charges will apply whenever you export the data from the cloud bucket to your local setup outside the Google Cloud infrastructure. This would apply in any case, whether you’re downloading from our “open-targets-genetics-releases” bucket, or from your own bucket to which you exported the BigQuery data.
It’s important to note that these are being charged by Google, so we have no control over that (and none of that money goes to Open Targets as well).
You can always fetch all of the data generated by Open Targets for free from the EMBL-EBI FTP server. Please see these links for further reference:
- How can I download data from the Open Targets Genetics Portal?
- Data Download - Open Targets Genetics Documentation
Finally, to address your question about automatic a sync. Unfortunately there isn’t a straightforward way to do this. Note that the schema of the data may (and frequently does) change between the releases, including adding and reorganising certian data fields. So it’s not an easy task to do a incremental sync in this situation.
Please let me know if you have any further questions, I’ll be happy to help