FTP Large File Download timeout Issue ~140 GB

Data Access Data downloads
, ,
1

We are unable to read/download the below file from FTP. Please find the details

Host : ftp.ebi.ac.uk

File path : /pub/databases/opentargets/genetics/latest/v2g/part-00367-3e2ec058-68cd-4c74-90dd-febdf43c76ff-c000.json

File Size : ~140GB

As the file size is around 140GB we are facing timeout issue after fetching nearly 30-40GB data. We have tried with a few ways to read/download the file but we are unable to process further.

Can you please help us on how to download this file? Or It will be helpful to fetch if the file is split into multiple files(each file ~ 10GB)

2

Hi Aparna

Without more details on what commands/tools you are using we cannot help you further. Timeouts are usually configurable, although it is possible that this is a problem on the EBI end too.

As of the upcoming release the data will be made available in Parquet format. JSON is not appropriate to this style of data as it produces very unwieldy file sizes.

3

Thank you @JarrodBaker for the update.
Please find the steps that we are following to download this file

  1. connect to FTP Server in Linux(passive mode)
    ftp ftp.ebi.ac.uk

  2. cd /pub/databases/opentargets/genetics/latest/v2g/

  3. get part-00367-3e2ec058-68cd-4c74-90dd-febdf43c76ff-c000.json

4

Hi JarrodBaker

May we know the next upcoming release date?