Ai+ Training

Reconciling data from disparate datasets can be a tricky and time-consuming process. Even when the data points refer to the same real-world entities, different data sources may use different conventions for describing their properties. For example, maintaining a global, up-to-date, and accurate dataset of infections and tests related to the COVID-19 pandemic is a challenging task, in part due to the different taxonomies that distinct nations and municipalities used to classify outcomes.

Ontologies are a simple solution to this problem. Ontologies are collections of class and relationship definitions. Data scientists can align disparate taxonomies with a centralized ontology - a “source of truth” for data classification, and unify their datasets in a consolidated hierarchy. As new datasets are added, they can be easily matched to the same ontology and reconciled with the existing data. Automating this process ensures that the datasets are unified in a consistent manner, and reduces the possibility of discrepancies arising from manual data curation. While manual taxonomy alignment may be easier in the short term, maintaining a process for ongoing taxonomy reconciliation is the only effective long-term solution.

In this session, we demonstrate how taxonomies from distinct datasets can be quickly reconciled and unified using a centralized ontology. As an example, we extract the taxonomies used in two open-source retail product datasets and align them with a common retail ontology. We also demonstrate the use of knowledge graph visualizations to showcase the impact of cross-dataset standardization. Finally, we discuss how this unification pipeline can be deployed at scale, using either open-source Python libraries or proprietary solutions like Neo4j.

Main learning points:

1. The importance of having a unified taxonomy across data sources and the difficulties involved in building that universal taxonomy

2. How to use ontologies to find common ground between disparate taxonomies to align them in a systematic and sustainable way

Local ODSC chapter in Boston, USA

Get your ODSC West 2021 pass with 75% OFF - https://bit.ly/2Rc9nRB