Hello, I have problems accessing the datasets on ftp.ebi.ac.uk via FTP (port 21).
I’m facing below issues
Error1:“425 Failed to establish connection".
Error2: “421 service not available remote service has closed”.
I expect to connect and download the datasets via FTP as it worked for me before.
Could you please help us on how to download the datasets through ftp connection.
Datasets to be downloaded:
2.Opentargets non genetics: /pub/databases/opentargets/platform/latest/
The EBI FTP server does have intermittent problems sometimes, so first of all I suggest that you check again if the problem has been resolved by now.
If that doesn’t work, could you share your OS (Linux/Windows/Mac) and which software are you using to fetch the files from the FTP?
@Kirill_Tsukanov From the past two weeks we are facing the issue . We tested again today but still facing the same issue.
Here are the OS details.
NAME=“Red Hat Enterprise Linux”
We are using FTPClient in linux for fetching files.
Could you please help us on how to download the files. Let us know if you need any further details.
Can you try downloading with wget and see if it works for you? For example:
wget -m ftp://ftp.ebi.ac.uk/pub/databases/opentargets/genetics/latest/
If it still doesn’t work, please send the error message the above command shows you.
We ran the below command
Command: wget -m ftp://ftp.ebi.ac.uk/pub/databases/opentargets/platform/latest/output/etl/ &> log_test.log
There are a total of 31404 files in the above mentioned folder but we can see that only 12000 files are downloaded using the above given command, remaining all files are failed with the error:
2023-10-23 23:01:34 (0.00 B/s) - Data connection: Connection timed out; Retrying.
--2023-10-23 23:01:35-- ftp://ftp.ebi.ac.uk/pub/databases/opentargets/platform/latest/output/etl/json/errors/evidence/sourceId%3Dimpc/part-00037-91841997-85c5-4805-89bc-636bcba235ae.c000.json
(try: 2) => ‘ftp.ebi.ac.uk/pub/databases/opentargets/platform/latest/output/etl/json/errors/evidence/sourceId=impc/part-00037-91841997-85c5-4805-89bc-636bcba235ae.c000.json’
==> CWD not required.
==> PASV ...
Cannot initiate PASV transfer.
--2023-10-23 23:01:35-- ftp://ftp.ebi.ac.uk/pub/databases/opentargets/platform/latest/output/etl/json/errors/evidence/sourceId%3Dimpc/part-00038-91841997-85c5-4805-89bc-636bcba235ae.c000.json
==> CWD not required.
==> PASV ...
Error in server response, closing control connection.
Can you please look into the previous comment.
Thank you for posting the error message.
Looking at the command you’re using, you don’t need to fetch the entire etl/ directory, because its subdirectories parquet/ and json/ contain exactly the same data, just in different formats. So as the first suggestion, try downloading just one of the subdirectories, depending on which format you want. Perhaps reducing the number of files will help with the reliability of the transfer.
Another flag which sometimes help inrease reliability is
--passive-ftp. This way, wget needs to open less connections for a download, also potentially increasing stability.
Taking the above suggestions into account, then, this is the command that I suggest you use (assuming you want “json” format; substitute it with “parquet” otherwise):
wget -m --passive-ftp ftp://ftp.ebi.ac.uk/pub/databases/opentargets/platform/latest/output/etl/json/ &> log_test.log
Please let me know if the same (or different) error occurs, and I’ll look into it further.
We ran the command that is provided in the previous comment.
wget -m --passive-ftp ftp://ftp.ebi.ac.uk/pub/databases/opentargets/platform/latest/output/etl/json/ &> log_test1.log
The job is still running since 14 days and 13298/15703 files succeedded which usually takes one day approximately for the job to complete all the files.
Hi @Vijay, I’m sorry to hear you are still having issues.
At this point I think we’ve tried most options we could. The FTP server is managed by EBI directly, rather than by Open Targets, so as your next step I suggest you contact EBI support: Support and feedback | EMBL-EBI — please explain your problem and the issues you had trying to download the data. (You can send a link to this post to avoid repeating all steps that we tried and the error messages.)
Sure @Kirill_Tsukanov will contact the EBI support team.
Thank you for helping us.
@Kirill_Tsukanov The EBI team is asking to raise ServiceNow ticket to technical team using below link.
But we don’t have any account with the mentioned ServiceNow portal.
Please find the below screenshot of login page.
We can’t find any create account option in the above login page.
Can you please help us in proceeding further.