Blocked during the download of files

Technical Support
1

Hi,

We try to download all the following files present in the site http://ftp.ebi.ac.uk using Python :

  • /pub/databases/opentargets/genetics/latest/d2v2g/
  • /pub/databases/opentargets/genetics/latest/l2g/
  • /pub/databases/opentargets/genetics/latest/search/variant/

but, we couldn’t download all the files all at once.

Here is what happens:

  1. The download of the files /pub/databases/opentargets/genetics/latest/d2v2g/ goes well until the file “part-00182-5fc32108-15fe-418e-9e53-52afa7c58284-c000.snappy.parquet”, and the script stops here (duration of the download : 1h57mn).
  2. If I stop the script, and retry the download, the following error occurred : “Connection timed out”
  3. If I wait 1 hour, and retry, the download of the files could continue. This time, I could download all the files of /pub/databases/opentargets/genetics/latest/l2g/ , and the files /pub/databases/opentargets/genetics/latest/search/variant/ up to “part-00033-867067f5-71c7-4946-bf6b-b1125d25951f-c000.json” (duration of the download : 49mn)
  4. Once again the download stops here

Thus, I would like to know what is happening. Were we temporarily blocked ? Are there any restrictions about the download of files ? In particular, is there a restriction about the number of downloads that can be done on a certain period of time ?
If this is the case, are there any way to download these files all at once ?

Thank you very much,

Kind regards

2

Hi @hoti,

Have you tried using lftp to download the files?

Best wishes,

Helena

3

Hi,

Thank you very much.

No we haven’t tried lftp yet. We will try it, and see if the same problem occurs or not.

Kind regards

4

Hi @hoti,

We have just been made aware by the EBI Technical Services that there are known issues with the EBI public FTP service. I will let you know when these are resolved, but they may explain why you have not been able to access the files.

You may want to check out one of our other data access options.

Best wishes,

Helena

5

Hi,

We have tried to use lftp, and it works really well. Thank you very much for that.

However, we still have a problem, but by trying something different that before. Here is what we tried, and what happened :

  1. We downloaded all the following files :
  1. We then tried to download all the following files :
  1. However, we could only download a few files of them, and then it failed. It appears that the website went down. We tried to access the very same site from a web browser, but it said that the website is down.
  2. We then waited a day, and could download the remaining files without problem

Could you tell us if the website went down just for me or for everyone ? Or, was the website not really down, but instead it blocked us the access as we had downloaded too much ?

If we were blocked, how much time should we wait before being able to download once again ?

Is there a way to use lftp so that it continues to try to download even if the site is down (until it comes back) ?

Thank you very much,

Kind regards

6

Hi @hoti,

I’m sorry you’re still experiencing issues! As this is a service run by EBI, I suggest you contact their technical team and let them know about this.

Best wishes,

Helena