Zum Hauptinhalt springen

Examples

You're welcome to copy and adapt the following examples at your convenience.

Application​

The application configuration contains static application settings, such as the locations of application directories and resources or global execution options.

▶ Example: application.yaml

connector:
config:
database:
# path to the working database file
file: /home/connector/db
dump:
# path to the dump directory
directory: ./temp
# dump filename template, store the dumps in separate directories for each service type
template: `${type}/${name}-${dump}-${date}T${time}`
service:
# path to the default service file
file: /home/connector/services.yaml
# path to the default service directory
directory: /home/connector/services
# scan the default service file and directory every 60 seconds
monitor:
interval: 60
placeholders:
# paths to additional properties files containing placeholders
files:
- ./resources/oauth.properties
- ./resources/secrets.properties
execution:
thread:
# number of threads used for parallel processing
count: 32
# maximum execution time for parallel processing, 60 seconds
timeout: 60000
# additional jdbc drivers
jars:
- type: jdbc
file: /home/connector/mssql-jdbc-12.10.0.jre11.jar
- type: jdbc
file: /home/connector/postgresql-42.7.5.jar

▶ Example: application.properties

# path to the working database file
connector.config.database.file=/home/connector/db

# path to the dump directory
connector.config.dump.directory=./temp

# dump filename template, store the dumps in separate directories for each service type
connector.config.dump.template=`${type}/${name}-${dump}-${date}T${time}`

# path to the default service file
connector.config.service.file=/home/connector/services.yaml

# path to the default service directory
connector.config.service.directory=/home/connector/services

# scan the default service file and directory every 60 seconds
connector.config.service.monitor.interval=60

# paths to additional properties files containing placeholders
connector.config.placeholders.files=./resources/oauth.properties,./resources/secrets.properties

# number of threads used for parallel processing
connector.config.execution.thread.count=32

# maximum execution time for parallel processing, 60 seconds
connector.config.execution.thread.timeout=60000

# additional jdbc drivers
connector.config.jars[0].type=jdbc
connector.config.jars[0].file=/home/connector/mssql-jdbc-12.10.0.jre11.jar
connector.config.jars[1].type=jdbc
connector.config.jars[1].file=/home/connector/postgresql-42.7.5.jar

Services​

DatabaseConnector​

▶ Example: DatabaseConnector - Microsoft SQL Server

The following service connects to a Microsoft SQL Server database with basic authentication and extracts metadata from specific schemas, before it uploads the transformed metadata to a destination application.

services:
MyService:
type: DatabaseConnector
# this section specifies the source database
source:
# connect to microsoft sql server database 'Sales'
url: jdbc:sqlserver://www.myserver.com:1433;DatabaseName=Sales;trustServerCertificate=true
# authenticate with username and password
authentication:
method: password
username: ${database.username}
password: ${database.password}
extensions:
# microsoft sql server extension to extract comments from extended property 'MS_Description'
Microsoft SQL Server:
comment: MS_Description
# this section specifies the extraction and transformation options
ingestion:
# schema filter
schemas:
# extract only these schemas
names:
accept:
- Finance
- Product
- KPI
# transform into collections with stereotype 'schema'
stereotype: schema
# table filter
tables:
# extract only these table types
types:
accept:
- TABLE
- VIEW
# transform into data objects with stereotype 'table'
stereotype: table
# this section specifies the destination application
upload:
# upload to my.dataspot.io into prod database
url: https://my.dataspot.io
database: prod
# the scheme must be a data model
scheme: Enterprise Model
# we're authenticating with client credentials (see below)
# therefore we need an access key to identify the service user
access-key: ${dataspot.access-key}
options:
# reconciliation mode
operation: REPLACE
# agent identification of this connector
agent-id: sales-connector
# authenticate with OAuth 2.0, using a client certificate (no password)
authentication:
method: oauth
provider-url: ${dataspot.oauth.provider-url}
client-id: ${dataspot.oauth.client-id}
credentials:
type: client-certificate
file: /home/connector/dataspot.pfx

The connection to Microsoft SQL Server will require a suitable JDBC driver. Be sure to download and specify the additional JAR in the application configuration:

connector:
config:
jars:
- type: jdbc
file: /home/connector/mssql-jdbc-12.10.0.jre11.jar

▶ Example: DatabaseConnector - PostgreSQL

The following service connects to a PostgreSQL database using additional properties to authenticate with a server/client certificate pair.

services:
MyService:
type: DatabaseConnector
# this section specifies the source database
source:
# connect to postgres database 'sales'
url: jdbc:postgresql://www.myserver.com:5432/sales
# authenticate with username and password...
authentication:
method: password
username: ${database.username}
password: ${database.password}
# ...but also pass the database-specific server/client certificate pair
properties:
sslmode: verify-full
sslrootcert: /home/connector/certs/server-ca.pem
sslcert: /home/connector/certs/client-cert.pem
sslkey: /home/connector/certs/client-key.pk8

# ingestion and upload options go here :)

The connection to PostgreSQL will require a suitable JDBC driver. Be sure to download and specify the additional JAR in the application configuration:

connector:
config:
jars:
- type: jdbc
file: /home/connector/postgresql-42.3.0.jar

▶ Example: DatabaseConnector - Google BigQuery

The following service connects to Google BigQuery using additional properties to authenticate with a keyfile.

services:
MyService:
type: DatabaseConnector
# this section specifies the source database
source:
# connect to google big query
url: jdbc:bigquery://https://www.googleapis.com/bigquery/v2:443;OAuthType=0
# the authentication properties could also be specified in the connection url
properties:
ProjectId: ${bigquery.projectId}
OAuthServiceAcctEmail: ${bigquery.service.account.email}
OAuthPvtKeyPath: /home/connector/bigquery-keyfile.json
ingestion:
# extract all projects and datasets (the JDBC driver maps projects to catalogs and datasets to schemas)
catalogs:
names:
accept: .*

The connection to Google BigQuery will require a suitable JDBC driver. Be sure to download and specify the additional JARs in the application configuration:

connector:
config:
jars:
- type: jdbc
directory: /home/connector/google-big-query # directory containing JDBC driver and 3rd party JARs

DatabricksConnector​

▶ Example: DatabricksConnector

The following service connects to a Databricks Unity Catalog instance, authenticates with a personal access token, and extracts metadata from a workspace, before it uploads the transformed metadata to a destination application.

services:
MyService:
type: DatabricksConnector
# this section specifies the databricks unity catalog instance
source:
url: https://dbc-00182f59-66ea.cloud.databricks.com
# the warehouse id is required to extract runtime data lineage
warehouse-id: b079313fa6222088
# authenticate with personal access token
authentication:
method: token
token: ${databricks.pat}
# this section specifies the extraction and transformation options
ingestion:
inCollection: Databricks
# catalog filter
catalogs:
# extract only this catalogs
names:
accept: my_workspace
# schema filter
schemas:
# extract these schemas, transform with different stereotypes
- names:
accept: gold
stereotype: gold
- names:
accept: silver
stereotype: silver
- names:
accept: bronze
stereotype: bronze
# extract runtime data lineage
lineages:
# extract runtime lineage since cutoff date
options:
cutoff-date: 2025-05-21
streams:
# extract column lineage
- type: column
filter:
# extract lineage create by a specific entity (e.g. a pipeline)
entity:
id: 939430187383108
# create transformations and rules
transformation:
label: Pipeline
stereotype: pipeline
rule:
stereotype: mapping
# this section specifies the destination application
upload:
# upload to my.dataspot.io into prod database
url: https://my.dataspot.io
database: prod
# the scheme must be a data model
scheme: Enterprise Model
# we're authenticating with client credentials (see below)
# therefore we need an access key to identify the service user
access-key: ${dataspot.access-key}
options:
# reconciliation mode and workflow statuses
operation: REPLACE
# agent identification of this connector
agent-id: databricks-connector
# authenticate with OAuth 2.0, using a client secret
authentication:
method: oauth
provider-url: ${dataspot.oauth.provider-url}
client-id: ${dataspot.oauth.client-id}
credentials:
type: client-secret
client-secret: ${dataspot.oauth.client-secret}

StorageConnector​

▶ Example: StorageConnector

The following service connects to an AWS S3 storage, authenticates with a secret key, and extracts metadata from specific file types, before it uploads the transformed metadata to a destination application.

services:
MyService:
type: StorageConnector
# this section specifies the source storage system
source:
# connect to aws s3 bucket
url: s3a://my-s3-bucket
# authenticate with access key and secret key
authentication:
method: secret-key
access-key: ${aws.s3.access-key}
secret-key: ${aws.s3.secret-key}
# use a specific aws region
properties:
fs.s3a.endpoint.region: eu-central-1
# this section specifies the extraction and transformation options
ingestion:
inCollection: AWS
options:
# process all objects added since the last run
read: since-last-run
directories:
# extract data objects from these paths
paths:
- sales/annual
- finance/*
files:
# partitioned parquet files
- type: parquet
partition:
delimiter: "="
# this section specifies the destination application
upload:
# upload to my.dataspot.io into prod database
url: https://my.dataspot.io
database: prod
# the scheme must be a data model
scheme: Enterprise Model
options:
# reconciliation mode and workflow statuses
operation: REPLACE
on-insert: WORKING
on-update: SUBMITTED
on-delete: INACTIVE
# agent identification of this connector
agent-id: s3-connector
# authenticate with username and password
authentication:
method: password
username: ${dataspot.basic.username}
password: ${dataspot.basic.password}

FileUpload​

▶ Example: FileUpload - Existing payload file

The following service uploads an existing payload file to a destination application.

services:
MyService:
type: FileUpload
source:
# use a placeholder to get the file path from an external source
file: ${payload.file}
upload:
url: https://my.dataspot.io
database: mydatabase
scheme: MyModel
# authenticate with username and password
authentication:
method: password
username: ${dataspot.basic.username}
password: ${dataspot.basic.password}

Start the CLI application to automatically upload a payload file, for example /home/connector/upload.json.gz:

java -Dpayload.file=/home/connector/upload.json.gz -jar dataspot-connector.jar --service=MyService

▶ Example: FileUpload - Download payload file

The following service downloads a payload file and then uploads it to a destination application.

services:
MyService:
type: FileUpload

before:
- label: Download payload file
exec: [curl, -o, /tmp/payload.json, https://example.com/payload.json]

- label: Validate payload file
exec: [jq, empty, /tmp/payload.json]

source:
file: /tmp/payload.json

upload:
url: https://my.dataspot.io
database: mydatabase
scheme: MyModel
# authenticate with username and password
authentication:
method: password
username: ${dataspot.basic.username}
password: ${dataspot.basic.password}

after:
- label: Delete payload file
exec: [rm, -f, /tmp/payload.json]

ApplicationReorg​

▶ Example: ApplicationReorg

The following service deletes expired jobs older than 2 weeks.

services:
MyService:
type: ApplicationReorg
job:
retention: 2w # delete jobs older than 2 weeks