DataHub
DataHub (opens in a new tab) is an open-source metadata platform for the modern data stack. It enables data discovery, data observability, and federated governance for your data ecosystem.
Prerequisites
In order to integrate with DataHub, you'll need to collect the following information:
| Name | Description |
|---|---|
| DataHub Server URL | The URL of your DataHub instance (e.g., http://localhost:8080 (opens in a new tab)) |
| Access Token | A DataHub access token with appropriate permissions. See DataHub Authentication (opens in a new tab) for more details. |
| BigQuery Credentials | Credentials that will be used on both Astralis and DataHub sides |
| Glossary Node | The node name in DataHub's glossary where terms will be created |
A Data Category in Astralis is represented as a Glossary Term in DataHub. We use these two terms interchangeably.
Every Glossary Term managed by Astralis is tied to a Glossary Term Group (named FidesDataCategories by default).
Datasets created by BigQuery are only supported at this time. This is verified by using the Dataset fides_meta.dataset.connection_type field.
How to setup
Setup on Astralis Side
- Create a new BigQuery integration
- Create a monitor for this integration
- Scan using this monitor
- Create a Dataset from the result of the scan
Setup on DataHub Side
- Create a new BigQuery ingestion
- Run the Ingestion
- Verify that the DataHub datasets were created
Configure Integration
- Create a new DataHub integration using the DataHub server connection info:
- DataHub Server URL
- DataHub Token
- Frequency (daily, weekly, monthly)
- Glossary Node Name
Once this has been set up, Astralis will sync the Data Categories according to the configured frequency.
How to use
Astralis is always actively checking in the background for Dataset/DataHub integration due to sync. This process:
- Runs once an hour
- Looks for due DataHub integrations to sync (using the last_run_timestamp and frequency fields)
- Processes related due Datasets to sync
- Will run a fresh DataHub integration at the next o'clock hour
Using the API for triggering a sync
You can trigger a DataHub sync using the following API endpoint:
POST /api/v1/plus/connection/datahub/{connection_key}/syncThe endpoint expects a request body that accepts a list of dataset_ids to sync. If the list is empty, nothing will be triggered. More details can be found in the Swagger UI.