Apache Airflow source configuration

Updated:

The Source configuration field in the technical lineage for Apache Airflow capability allows you reduce the amount of data objects to be processed and enhance the performance of CollibraData Lineage.

The value of the Source configuration field must be a valid block of JSON code, for example:

{
    "datasources": [
        {
            "namespace": "s3a://my-dev",
            "group": "file_group_my-dev",
            "type": "file"
        },
        {
            "namespace": "s3a://my-dev2",
            "group": "file_group_my-dev2",
            "type": "file"
        },
        {
            "namespace": "snowflake://myorg.snowflake.com",
            "type": "database",
            "collibraSystemName": "snowflake",
            "database": "snowflake_db",
            "schema": "snowflake_db",
            "dialect": "snowflake"
        }
    ]
}

The following table describes the various properties you can use in your JSON code block.

Property

 Description Required?
datasources

An array of mappings for Airflow namespaces to Collibra system names and databases. This section includes the properties to translate the system name, database, schema, and dialect.

No

namespace

The namespace that is used by Airflow. The value of this property must match the namespace in the Airflow OpenLineage files.

Yes

group

Specifies a group name used to organize files in the namespace.

Specify this property only when the type property is set to file. Otherwise, the system uses the namespace value as the file group.

No

type

The type of data source that this namespace contains. Specify one of the following values:

  • database.
  • file.

If you do not specify this property, Collibra Data Lineage derives the value from the JSON schema. If you specify this property, your provided value takes precedence.

When this property is set to file, you must specify the group property.

Yes

collibraSystemName

The system or server name of the data source.

Use this property with the useCollibraSystemName property in the lineage harvester configuration file to override the default Collibra System asset name for this data source.

Specify this property with the same name as the name of the System asset that you create when you prepare the physical data layer in Data Catalog. If you don't prepare the physical data layer, Collibra Data Lineage cannot stitch the data objects in your technical lineage to the assets in Data Catalog.

The following rules apply when you specify the system name:

  • If you do not specify this property, it is set to DEFAULT.
  • This property is used only when the type property is set to database.
  • This property could assist in stitching to Catalog assets.

No

database

The name of the default database that the namespace connection refers to.

The following rules apply when you specify the database name:

  • The value of this property is determined in the following order of precedence:
    1. The value you specify.
    2. If not specified, Collibra Data Lineage derives the value from the namespace and name fields based on the OpenLineage naming convention.
    3. If the value cannot be derived, Collibra Data Lineage uses database names defined in queries or output dataset names.
  • This property is used only when the type property is set to database.
  • This property could assist in stitching to Catalog assets.

No

schema

The name of the default schema, to be used with the namespace connection.

The following rules apply when you specify the schema name:

  • The value of this property is determined in the following order of precedence:
    1. The value you specify.
    2. If not specified, Collibra Data Lineage derives the value from the namespace and name fields based on the OpenLineage naming convention.
    3. If the value cannot be derived, Collibra Data Lineage uses schema names defined in queries or output dataset names.
  • This property is used only when the type property is set to database.
  • This property could assist in stitching to Catalog assets.

No

dialect

When no columnLineage is present, Collibra Data Lineage tries to parse any SQL present. Set the dialect to parse SQL properly.

You can enter one of the following values:

  • azure, for an Azure SQL Server data source.
  • bigquery, for a Google BigQuery data source.
  • db2, for an IBM DB2 data source.
  • hana, for an SAP HANA data source.
  • hana-cviews, for getting lineage from calculated views in an SAP HANA Classic on-premises data source.
  • hana-cviews-v2, for getting lineage from calculated views in an SAP HANA Cloud/Advanced data source.
    Important 

    To get technical lineage including calculated views, you must harvest SAP HANA by adding two Technical Lineage for SqlDirectory capabilities with the Shared Storage connections. In one capability, specify the hana dialect, and in the other, specify the hana-cviews or hana-cviews-v2 dialect.

  • hive, for a HiveQL data source.
  • greenplum, for a Greenplum data source.
  • mssql, for a Microsoft SQL Server data source.
  • mysql, for a MySQL data source.
  • netezza, for a Netezza data source.
  • oracle, for an Oracle data source.
  • postgres, for a PostgreSQL data source.
  • redshift, for an Amazon Redshift data source.
  • snowflake, for a Snowflake data source.
  • spark, for a Spark SQL data source.
  • sybase, for a Sybase data source.
  • teradata, for a Teradata data source.

No