# Older versions of OpenTargets data

**URL:** <https://community.opentargets.org/t/older-versions-of-opentargets-data/1541>\
**Category:** General\
**Tags:** ot-platform\
**Created:** [1 September 2024 14:16 UTC](https://community.opentargets.org/t/older-versions-of-opentargets-data/1541 "2024-09-01T14:16:13Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![sanjayb100](https://avatars.discourse-cdn.com/v4/letter/s/cc9497/32.png) [@sanjayb100](https://community.opentargets.org/u/sanjayb100)\
**Post date:** [1 September 2024 14:16 UTC](https://community.opentargets.org/t/older-versions-of-opentargets-data/1541/1 "2024-09-01T14:16:13Z")

</div>

A question regarding archived versions of OpenTargets data. The oldest version of OpenTargets data that I can see is from July 2019:  
[https://ftp.ebi.ac.uk/pub/databases/opentargets/platform/](https://ftp.ebi.ac.uk/pub/databases/opentargets/platform/)  
I wondered if there were older versions of OpenTargets data available elsewhere or on request, and if so, what is the oldest available collated data?

Many thanks,

Sanjay

---

<div class="post-metadata">

**Author:** ![dsuveges](https://dub1.discourse-cdn.com/flex017/user_avatar/community.opentargets.org/dsuveges/32/30_2.png) [@dsuveges](https://community.opentargets.org/u/dsuveges)\
**Post date:** [2 September 2024 09:06 UTC](https://community.opentargets.org/t/older-versions-of-opentargets-data/1541/2 "2024-09-02T09:06:50Z")

</div>

Hi Sanjay, Under the ftp address you have pasted, there is a complete collection of OpenTargets Platform data releases. The earliest releases from 16.04. However, keep in mind that over the years the data model changed significantly, which can make a systemic comparison relatively complicated.

Best,  
Daniel

---

<div class="post-metadata">

**Author:** ![ochoa](https://dub1.discourse-cdn.com/flex017/user_avatar/community.opentargets.org/ochoa/32/189_2.png) [@ochoa](https://community.opentargets.org/u/ochoa)\
**Post date:** [2 September 2024 09:18 UTC](https://community.opentargets.org/t/older-versions-of-opentargets-data/1541/3 "2024-09-02T09:18:33Z")

</div>

@sanjayb100 you might want to look at how @eczech and colleagues performed a temporal analysis on the Open Targets data.

> **[Clinical Advancement Forecasting](https://www.medrxiv.org/content/10.1101/2024.08.02.24311422v2)**
>
> Choosing which drug targets to pursue for a given disease is one of the most impactful decisions made in the global development of new medicines. This study examines the extent to which the outcomes of clinical trials can be predicted based on a...

---

<div class="post-metadata">

**Author:** ![eczech](https://dub1.discourse-cdn.com/flex017/user_avatar/community.opentargets.org/eczech/32/416_2.png) [@eczech](https://community.opentargets.org/u/eczech)\
**Post date:** [2 September 2024 16:59 UTC](https://community.opentargets.org/t/older-versions-of-opentargets-data/1541/4 "2024-09-02T16:59:24Z")

</div>

> you might want to look at how @eczech and colleagues performed a temporal analysis on the Open Targets data.

It wasn’t in that paper, but we have built some consolidated datasets across OT versions going all the way back to 16.04 (i.e. from 2016). We weren’t merging schemas for raw evidence across years extensively, so we were able to eschew most of the problems @dsuveges mentioned. We created a simple, merged view like this: `ot_version`, `gene_id`, `diesase_id`, `datasource_id`, `score`. Here is an example of the schemas/data we were merging:

```python
! curl http://ftp.ebi.ac.uk/pub/databases/opentargets/platform/16.04/16.04_association_data.json.gz \
| gzip -dc | head -n 1000 > /tmp/16.04_association_data.sample.json
(
    spark.read.json("/tmp/16.04_association_data.sample.json")
    .select(
        F.col("target.id").alias("gene_id"),
        F.col("disease.id").alias("disease_id"),
        F.col("association_score.datasources.*")
    )
    .transform(lambda df: (
        df.select(
            "gene_id", "disease_id", 
            F.array(*[
                F.struct(F.lit(c).alias("datasource_id"), F.col(c).alias("score"))
                for c in df.columns if c not in {"gene_id", "disease_id"}
            ]).alias("scores")
        )
    ))
    .select("*", F.explode("scores").alias("score"))
    .select("gene_id", "disease_id", "score.datasource_id", "score.score")
    .printSchema()
)
root
 |-- gene_id: string (nullable = true)
 |-- disease_id: string (nullable = true)
 |-- datasource_id: string (nullable = false)
 |-- score: double (nullable = true)

```

Here are a few other things to watch out for in doing this:

1. Older datasets have an `is_direct` flag in the data to delineate between direct associations and those attained through EFO ancestors. Where that is true, you get the equivalent of the more recent “direct” datasets at [Open Targets Downloads](https://platform.opentargets.org/downloads). Where that is **either** true or false, you get the equivalent of the “indirect” datasets in the more recent downloads. That might not be obvious at first, so it’s worth knowing if you’re trying to work across versions.
2. Some of the folder structures change over time. Starting with version 18.12, data starts appearing in `output` directories so you’ll need a switch for directory layout like that.
3. I believe older datasets included associations with a score of 0 while newer ones don’t, so you’ll likely want to filter out 0 scores or somehow deal with that inconsistency.

---

<div class="post-metadata">

**Author:** ![sanjayb100](https://avatars.discourse-cdn.com/v4/letter/s/cc9497/32.png) [@sanjayb100](https://community.opentargets.org/u/sanjayb100)\
**Post date:** [10 September 2024 17:15 UTC](https://community.opentargets.org/t/older-versions-of-opentargets-data/1541/5 "2024-09-10T17:15:58Z")

</div>

Thanks all! I hadn’t seen that preprint, I’ll be reading this in detail! Thanks also for the pointers on some of the practical challenges in this approach