Datahub

DataHub is an open-source metadata platform for the data stack. DataHub is a modern data catalog built to enable end-to-end data discovery, data observability, and data governance. It supports various data sources including PostgreSQL.

Because YugabyteDB's YSQL API is wire-compatible with PostgreSQL, Datahub can connect to YugabyteDB as a data source using the PostgreSQL plugin.

Setup

You can run the Docker Compose quickStart example provided in the Datahub GitHub repository against YugabyteDB with the following changes:

• Replace the MySql Docker image with that of YugabyteDB.
• Specify the entrypoint command for the YugabyteDB Docker container.
• Change port from 5432 to 5433
• Change username and password to yugabyte.
• Change the driver to org.postgresql.Driver.

Make changes in the following files:

• In docker/quickstart/docker-compose-without-neo4j.quickstart.yml, change the following:

  • Change the EBEAN_DATASOURCE configuration [lines 80-84 and 126-130] as follows:

    EBEAN_DATASOURCE_DRIVER=org.postgresql.Driver
    EBEAN_DATASOURCE_HOST=yugabyte:5433
    EBEAN_DATASOURCE_PASSWORD=yugabyte
    EBEAN_DATASOURCE_URL=jdbc:postgresql://yugabyte:5433/yugabyte
    EBEAN_DATASOURCE_USERNAME=yugabyte
    

  • Change mysql-setup to postgres-setup [line 123].

  • Replace the mysql and mysql-setup container [lines 197 - 231] with yugabyte and postgres-setup container as follows:

    yugabyte:
       container_name: yugabyte
       hostname: yugabyte
       image: yugabytedb/yugabyte:latest
       command: /bin/bash /home/yugabyte/docker-entrypoint-initdb.d/yb-init.sh
       environment:
         POSTGRES_USER: ${POSTGRES_USER:-yugabyte}
         POSTGRES_PASSWORD: ${POSTGRES_PASSWORD:-yugabyte}
       ports:
       - '5433:5433'
       volumes:
       - ./yb-setup/:/home/yugabyte/docker-entrypoint-initdb.d/
       healthcheck:
         test: bin/ysqlsh -h `hostname -i` -U yugabyte -tAc 'select 1' -d yugabyte
         interval: 10s
         timeout: 5s
         retries: 20
    postgres-setup:
      container_name: postgres-setup
      depends_on:
        yugabyte:
          condition: service_healthy
      environment:
      - POSTGRES_HOST=yugabyte
      - POSTGRES_PORT=5433
      - POSTGRES_USERNAME=yugabyte
      - POSTGRES_PASSWORD=yugabyte
      - DATAHUB_DB_NAME=yugabyte
      hostname: yugabyte-setup
      image: ${DATAHUB_POSTGRES_SETUP_IMAGE:-acryldata/datahub-postgres-setup}:${DATAHUB_VERSION:-head}
    

• Create a directory yb-setup in docker/quickstart/ and a script file named yb-init.sh with the following content and place it under docker/quickstart/yb-setup/ in the repository. The script runs during container initialization to launch the YugabyteDB cluster.

  bin/yugabyted start
  
  sleep 5
  
  bin/ysqlsh -h `hostname -i` -f /home/yugabyte/docker-entrypoint-initdb.d/init.sql
  tail -f /dev/null
  

• Copy the file docker/postgres/init.sql to docker/quickstart/yb-setup/.

Run the example

Run the example using the following command:

docker compose -f docker-compose-without-neo4j.quickstart.yml up -d

After all the containers are running, you can ingest some demo data by running ./datahub/docker/ingestion/ingestion.sh, or head to http://localhost:9002 (username: datahub, password: datahub) to access the UI.