# Difference between parquet files and website/API

**URL:** <https://community.opentargets.org/t/difference-between-parquet-files-and-website-api/1349>\
**Category:** Data issue\
**Tags:** data, data-updates\
**Created:** [7 February 2024 10:21 UTC](https://community.opentargets.org/t/difference-between-parquet-files-and-website-api/1349 "2024-02-07T10:21:46Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![fru](https://avatars.discourse-cdn.com/v4/letter/f/74df32/32.png) [@fru](https://community.opentargets.org/u/fru)\
**Post date:** [7 February 2024 10:21 UTC](https://community.opentargets.org/t/difference-between-parquet-files-and-website-api/1349/1 "2024-02-07T10:21:46Z")

</div>

Hi,

I have downloaded the **parquet** files for **“Associations - direct (by data type)”** ([Open Targets Platform](https://platform.opentargets.org/downloads)). I have extracted information for a specific disease (MONDO\_0005277). When I compare the parquet files with the data in the website ([Open Targets Platform](https://platform.opentargets.org/disease/MONDO_0005277/classic-associations)) and the data retrieved using the API, I see some differences. The API and the website agree on the number of associated targets and contain the same information but the parquet files seem to miss some data. In particular, they do not contain info on the **“animal\_model”** data type.

Are the parquet files not updated?

**Command line to download the data:**  
`wget --recursive --no-parent --no-host-directories --cut-dirs 8 ftp://ftp.ebi.ac.uk/pub/databases/opentargets/platform/23.12/output/etl/parquet/associationByDatatypeDirect`

**R code to extract the information for the specific disease:**

```auto
library(dplyr)
library(sparklyr)
library(sparklyr.nested)

## establish connection
sc <- spark_connect(master = "local")

## read dataset
evd <- spark_read_parquet(sc, path = path)

## Browse the schema
columns <- evd %>%
  sdf_schema() %>%
  lapply(function(x) do.call(tibble, x)) %>%
  bind_rows()

## select fields of interest
evdSelect <- evd %>%
  select(diseaseId,
         targetId,
         datatypeId,
         score,
         evidenceCount) 

evdSelect_disease <- evdSelect %>%
  collect() %>%
  dplyr::filter(diseaseId == "MONDO_0005277")

unique(evdSelect_disease$datatypeId)
# [1] "literature" "known_drug" "genetic_association"

```

In my hands, the “animal\_model” data type is missing but it’s contained in the website:

 ![image](https://europe1.discourse-cdn.com/flex017/uploads/opentargets/original/1X/154edbdc3aebb8d7aaa6d8dd76671b6bdc3d4a32.png)

Thanks for your feedback!

Francesco

---

<div class="post-metadata">

**Author:** ![hcornu](https://dub1.discourse-cdn.com/flex017/user_avatar/community.opentargets.org/hcornu/32/716_2.png) [@hcornu](https://community.opentargets.org/u/hcornu)\
**Post date:** [7 February 2024 10:33 UTC](https://community.opentargets.org/t/difference-between-parquet-files-and-website-api/1349/2 "2024-02-07T10:33:20Z")

</div>

Hi @fru, and welcome to the Open Targets Community! 🎉

I think this is probably because you are looking at the `Associations — direct` dataset, whereas the data for animal models for Migraine Disorder appears to be **indirect evidence** , mainly for `familial or sporadic hemiplegic migraine` ([MONDO\_0018925](https://www.ebi.ac.uk/ols4/ontologies/efo/terms?short_form=MONDO_0018925)), which is a child term of Migraine Disorder.

You can find more information about direct/indirect evidence and how it’s show in the Platform [in our documentation](https://platform-docs.opentargets.org/associations#ontological-selection-of-evidence).

Let me know if you have any questions about this!

Best wishes,

Helena

---

<div class="post-metadata">

**Author:** ![fru](https://avatars.discourse-cdn.com/v4/letter/f/74df32/32.png) [@fru](https://community.opentargets.org/u/fru)\
**Post date:** [7 February 2024 11:31 UTC](https://community.opentargets.org/t/difference-between-parquet-files-and-website-api/1349/3 "2024-02-07T11:31:23Z")

</div>

Thanks for the reply! I see, it makes sense. I guess the file **“Associations - indirect (by data type)”** should contain the indirect evidence for Migraine Disorder as well as it’s stated in the website and not only reported for the child term 🙂 Anyway, your reply fixes my problem, thanks again!

Francesco

---

<div class="post-metadata">

**Author:** ![fru](https://avatars.discourse-cdn.com/v4/letter/f/74df32/32.png) [@fru](https://community.opentargets.org/u/fru)\
**Post date:** [7 February 2024 13:12 UTC](https://community.opentargets.org/t/difference-between-parquet-files-and-website-api/1349/4 "2024-02-07T13:12:47Z")

</div>

For future reference, the **“Associations - indirect (by data type)”** file contains the same number of targets and associations as the website and API. It looks like it contains both **direct** and **indirect** associations: **“literature”, “known\_drug”, “genetic\_association”, “animal\_model”**.  
The only data types not present in this file are: **“somatic\_mutations”, “affected\_pathway”, “rna\_expression”** because those are actually not available for the disease ID “MONDO\_0005277”, but these data types are present in **“Associations - indirect (by data type)”**.

Francesco