# Questions about the 22.06 dataset — overlap in disease terms/multiple entries for disease-gene associations **URL:** \ **Category:** General\ **Created:** [6 July 2022 15:43 UTC](https://community.opentargets.org/t/questions-about-the-22-06-dataset-overlap-in-disease-terms-multiple-entries-for-disease-gene-associations/698 "2022-07-06T15:43:44Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![vcatterson](https://avatars.discourse-cdn.com/v4/letter/v/edb3f5/32.png) [@vcatterson](https://community.opentargets.org/u/vcatterson)\ **Post date:** [6 July 2022 15:43 UTC](https://community.opentargets.org/t/questions-about-the-22-06-dataset-overlap-in-disease-terms-multiple-entries-for-disease-gene-associations/698/1 "2022-07-06T15:43:44Z")
Hello! I have a couple of **questions about the newest version of the dataset** , and specifically the version hosted on S3 as part of the RODA. We have been using the previous version of the dataset for a while now, and recently started migrating to v22. We’re also transitioning from using the FTP server to using the S3 bucket. In these transitions, we’ve noticed changes in the data which are hard to understand. A couple of examples are below. ## More disease entries, but more overlap in terms In the diseases list accessed from s3://aws-roda-hcls-datalake/opentargets\_latest/diseases/, there are now more entries than in the previous version, but it seems that the same entry is included multiple times with slightly different data. For example: - Orphanet\_217607 is included in 5 rows. 4 rows appear to be exact duplicates of each other. - The 5th row for Orphanet\_217607 has a different dbXRefs value, and one fewer parent in the parents column. - Orphanet\_217607 is “Familial dilated cardiomyopathy”. There is also a row for “familial dilated cardiomyopathy” (all lower case) with ID MONDO\_0016333. We didn’t see this sort of overlap in naming in the previous version. ## Multiple entries for disease-gene associations in the S3 bucket In the dataset which associates diseases with genes accessed at s3://aws-roda-hcls-datalake/opentargets\_latest/associationbydatatypeindirect/, there are multiple entries for a given tuple of \. Specifically there are 5 entries for the tuple of: diseaseId: Orphanet\_217607 targetId: ENSG00000160789 datatypeID: genetic\_association The 5 rows all differ in score and evidenceCount, and it is not clear why these differences exist. In the version of this data accessed using wget there is a single entry per tuple, which is what we expect. The wget command is: `` ## Questions Our main question is how to reconcile these seemingly duplicate entries which have the same ID but different data. If you don’t have suggestions for that, it would be helpful to understand more about why these changes were made for v22 and/or S3 (which might give us direction on how to reconcile the entries). Thanks in advance for any advice you can give. ---
**Author:** ![ochoa](https://dub1.discourse-cdn.com/flex017/user_avatar/community.opentargets.org/ochoa/32/189_2.png) [@ochoa](https://community.opentargets.org/u/ochoa)\ **Post date:** [7 July 2022 09:07 UTC](https://community.opentargets.org/t/questions-about-the-22-06-dataset-overlap-in-disease-terms-multiple-entries-for-disease-gene-associations/698/2 "2022-07-07T09:07:17Z")
> More disease entries, but more overlap in terms There is an issue in the current version of the Experimental Factor Ontology that the Platform uses causing issues with the Orphanet IDs. The observed behaviour is that a significant number of terms with Orphanet ID are now also included in MONDO. The “familial dilated cardiomyopathy” is a good example. There is a ticket to follow up on the issue: > > > At least in 3.42 and 3.43, there are a large number of duplicated terms in EFO m…ostly affecting rare diseases. > > Just by lower-casing the names and looking for exact matches, there are 3036 duplicated terms (v3.42). Some of them are explained by disease vs phenotype conondrum, but the vast majority correspond to a MONDO vs Orphanet duplication. > > Some examples: > > \[Hemophilia\](https://www.ebi.ac.uk/ols/ontologies/efo/terms?iri=http%3A%2F%2Fwww.orpha.net%2FORDO%2FOrphanet\_448) Orphanet:448 - \[hemophilia\](https://www.ebi.ac.uk/ols/ontologies/efo/terms?iri=http%3A%2F%2Fpurl.obolibrary.org%2Fobo%2FMONDO\_0018660) MONDO:0018660 > \[Fragile X syndrome\](https://www.ebi.ac.uk/ols/ontologies/efo/terms?iri=http%3A%2F%2Fwww.orpha.net%2FORDO%2FOrphanet\_908) Orphanet:908 - \[fragile X syndrome\](https://www.ebi.ac.uk/ols/ontologies/efo/terms?iri=http%3A%2F%2Fpurl.obolibrary.org%2Fobo%2FMONDO\_0010383) MONDO:0010383 > \[Apert syndrome\](https://www.ebi.ac.uk/ols/ontologies/efo/terms?iri=http%3A%2F%2Fwww.orpha.net%2FORDO%2FOrphanet\_87) Orphanet:87 - \[apert syndrome\](https://www.ebi.ac.uk/ols/ontologies/efo/terms?iri=http%3A%2F%2Fpurl.obolibrary.org%2Fobo%2FMONDO\_0007041) MONDO:0007041 > > ... > Multiple entries for disease-gene associations in the S3 bucket Based on your observation, it looks like there is a mismatch in the dataset that S3 is serving. Very likely, this dataset does not really correspond to “associationByDatatypeIndirect”. Very likely this is another association aggregation (e.g. associationByDatasourceIndirect). We don’t actively move the data to S3, so we will work with AWS to resolve the problem. Thanks for flagging this. In the meantime, I would recommend downloading the dataset directly from the EBI FTP as this is supposed to be the source of truth. ---
**Author:** ![vcatterson](https://avatars.discourse-cdn.com/v4/letter/v/edb3f5/32.png) [@vcatterson](https://community.opentargets.org/u/vcatterson)\ **Post date:** [8 July 2022 19:35 UTC](https://community.opentargets.org/t/questions-about-the-22-06-dataset-overlap-in-disease-terms-multiple-entries-for-disease-gene-associations/698/3 "2022-07-08T19:35:00Z")
Thank you so much for these replies! Very helpful to understand more about the underlying issues, and to know that they are both being followed-up on. (I am the OP, but I only just created my account and Helpdesk very kindly posted this query for me.) I really appreciate all the work that you and the team do, and I’m grateful for your suggestions of work-arounds, too. ---
**Author:** ![dsuveges](https://dub1.discourse-cdn.com/flex017/user_avatar/community.opentargets.org/dsuveges/32/30_2.png) [@dsuveges](https://community.opentargets.org/u/dsuveges)\ **Post date:** [9 July 2022 17:19 UTC](https://community.opentargets.org/t/questions-about-the-22-06-dataset-overlap-in-disease-terms-multiple-entries-for-disease-gene-associations/698/4 "2022-07-09T17:19:37Z")
Hi @vcatterson , We have contacted the AWS team to clarify the duplication issue with the datasets they fetch from the EBI ftp. More details under [this ticket](https://github.com/opentargets/issues/issues/2661). We are waiting for their reply. In the meantime, I would suggest to source from the EBI ftp. Best, Daniel