Skip to main content

Installation

Introduction

To use the Tokern Lineage engine, you need to:

  • Deploy the engine with access to a Postgres database.
  • Set up and test integration libraries with your favorite ETL tools like Apache Airflow or dbt.
  • Browse lineage graphs or programmatically analyze graphs in Python notebooks.

Architecture

Tokern Lineage Engine consists for the following components:

  • API Server
  • Kedro Viz
  • API SDK
  • DB SDK

Install using packages

Setup the postgres database

# Create database and role in psql prompt
CREATE DATABASE tokern;
CREATE ROLE tokern_admin WITH LOGIN PASSWORD '<secret password>';
GRANT ALL PRIVILEGES ON DATABASE tokern TO tokern_admin;

Setup the API Server

python3 -m venv .env
source .env/bin/activate
pip install data-lineage

data_lineage --help
Usage: data_lineage [OPTIONS]

Options:
--version                       Show the version and exit.
-l, --log-level TEXT            Logging Level
--catalog-user TEXT             Database user name  [required]
--catalog-password TEXT         Database Password  [required]
--catalog-host TEXT             Database Host
--catalog-port INTEGER          Database Password
--catalog-db TEXT               Postgres Database
--server-address TEXT           The socket to bind to
--is-production / --not-production
                                Run server in development mode
--help                          Show this message and exit.

# Start the server
data_lineage --catalog-user tokern_admin --catalog-password <password>

Setup Kedro Viz

git clone https://github.com/tokern/kedro-viz.git
cd kedro-viz
npm start

Install using Docker

This guide will help you get the Tokern Lineage engine and Postgres running as Docker containers using Docker Compose. This is the easiest way to set up Tokern Lineage engine on your local environment.

Prerequisites

Download and start services

# in a new directory run
wget https://raw.githubusercontent.com/tokern/data-lineage/master/install-manifests/docker-compose/tokern-lineage-engine.yml
# or run
curl https://raw.githubusercontent.com/tokern/data-lineage/master/install-manifests/docker-compose/catalog-demo.yml -o tokern-lineage-engine.yml

Run docker-compose

docker-compose up -d

If you want to use an external Postgres database, change the following parameters in tokern-lineage-engine.yml:

  • CATALOG_HOST
  • CATALOG_USER
  • CATALOG_PASSWORD
  • CATALOG_DB

You can also override default values using environement variables.

CATALOG_HOST=... CATALOG_USER=... CATALOG_PASSWORD=... CATALOG_DB=... docker-compose -f ... up -d

For more advanced usage of environment variables with docker-compose, refer to docker-compose docs

Head to http://localhost:8000/ to open the Tokern Lineage app