Skip to main content

Query Parsing


This example demonstrates how to scan query history from a data warehouse and save it in the Tokern Lineage engine. The app automatically parses and extracts data lineage from the queries.

The example consists of the following sequence of operations:

  • Start docker containers containing a demo. Refer to docs for detailed instructions on installing demo-wikimedia.
  • Scan and send queries from query history to data lineage app.
  • Visualize the graph by visiting Tokern UI.
  • Analyze the graph

Download queries for the wikimedia demo from Github

Install Wikimedia demo

This demo requires wikimedia demo to be running. Start the demo using the following instructions:

# in a new directory run
# or run
curl -o docker-compose.yml

Run docker-compose

docker-compose up -d

Verify containers are running

docker container ls | grep tokern
# Required configuration for API and wikimedia database network address

docker_address = ""
wikimedia_db = {
"username": "etldev",
"password": "3tld3v",
"uri": "tokern-demo-wikimedia",
"port": "5432",
"database": "wikimedia"
from data_lineage import Catalog

catalog = Catalog(docker_address)

# Register wikimedia datawarehouse with data-lineage app.

source = catalog.add_source(name="wikimedia", source_type="postgresql", **wikimedia_db)

# Scan the wikimedia data warehouse and register all schemata, tables and columns.

# Read queries from a json file. Note that this is for demo only

with open("queries.json", "r") as file:
queries = json.load(file)
from datetime import datetime
from data_lineage import Analyze

analyze = Analyze(docker_address)

for query in queries:
analyze.analyze(**query, source=source,,

Visit Kedro UI

One Task Graph