DataStage source configuration

Updated:

Note This topic is only relevant if you are creating technical lineage via Edge. If you are using the CLI lineage harvester (deprecated), you need to create a <source ID> configuration file. The CLI harvester will officially reach its End of Life on July 31, 2026.

The Source configuration field in the DataStage technical lineage Edge capability allows you to:

The value of the Source configuration field must be a valid block of JSON code, for example:

Copy 
{
  "OdbcDataSources": {
    "oracle-data-source": {
      "dbname": "my-oracle-database",
      "schema": "my-oracle-schema",
      "dialect": "oracle",
      "collibraSystemName": "my-system"
    },
    "mssql-data-source": {
      "dbname": "my-mssql-database",
      "schema": "my-mssql-schema",
      "dialect": "mssql",
      "collibraSystemName": "my-system"
    }
  },
  "NonOdbcConnectors": {
    
    "admin@database-name": {
      "dbname": "my-netezza-database",
      "schema": "my-netezza-schema",
      "dialect": "netezza",
      "collibraSystemName": "my-system"
    },
    "admin@second-database-name": {
      "dbname": "my-second-netezza-database",
      "schema": "my-second-netezza-schema",
      "dialect": "netezza",
      "collibraSystemName": "my-system"
    }
  },   
  "jobs": [
    "my_job_1",
    "my_job_2"
  ],
  "jobParameters": [
    {
      "name": "parameter_value_1"
      "value": "parameter_value_1"
    },
    {
      "name": "parameter_name_2",
      "value": "parameter_value_2"
    }
  ]
  "perJobParameters": {
      "jobId1": [
          {
              "name": "parameter_name_1",
              "value": "parameter_value_3"
          }
       ],
       "jobId2": [
           {
               "name": "parameter_name_1",
               "value": "parameter_value_4"
           }
       ]
  }
}

The following table describes the various properties you can use in your JSON code block.

Property

 Description

OdbcDataSources

Open Database Connectivity data sources in IBM InfoSphere DataStage for which you want to create a technical lineage.
<data-source-name>

The ODBC data source name that you use in your DataStage projects.

This section contains the properties to translate the database, schema and dialect.
dbname
 The name of your database, to which the ODBC data source connection refers.
schema

The name of your schema, to which the ODBC data source connection refers.
dialect

The dialect of the referenced database.

You can enter one of the following values:

  • azure, for an Azure SQL Server data source.
  • bigquery, for a Google BigQuery data source.
  • db2, for an IBM DB2 data source.
  • hana, for an SAP HANA data source.
  • hana-cviews, for getting lineage from calculated views in an SAP HANA Classic on-premises data source.
  • hana-cviews-v2, for getting lineage from calculated views in an SAP HANA Cloud/Advanced data source.
    Important 

    To get technical lineage including calculated views, you must harvest SAP HANA by specifying two data sources in the lineage harvester configuration file. In one data source, specify the hana dialect, and in the other, specify the hana-cviews or hana-cviews-v2 dialect.

  • hive, for a HiveQL data source.
  • greenplum, for a Greenplum data source.
  • mssql, for a Microsoft SQL Server data source.
  • mysql, for a MySQL data source.
  • netezza, for a Netezza data source.
  • oracle, for an Oracle data source.
  • postgres, for a PostgreSQL data source.
  • redshift, for an Amazon Redshift data source.
  • snowflake, for a Snowflake data source.
  • spark, for a Spark SQL data source.
  • sybase, for a Sybase data source.
  • teradata, for a Teradata data source.
collibraSystemName

The system or server name of the data source.

Use this property with the useCollibraSystemName property in the lineage harvester configuration file to override the default Collibra System asset name for this data source.

Specify this property with the same name as the name of the System asset that you create when you prepare the physical data layer in Data Catalog. If you don't prepare the physical data layer, Collibra Data Lineage cannot stitch the data objects in your technical lineage to the assets in Data Catalog.

This property is optional.

NonOdbcConnectors

Other data source connectors in IBM InfoSphere DataStage for which you want to create a technical lineage. For example, DB2, Oracle or Netezza.

Note This section is optional.
<data-source-connector-ID>

The data source username and database of the connector that you use in your DataStage projects. This usually looks like for example admin@database-name. The combination of the username and database name should be unique.

The following section contains the properties to translate the database, schema and dialect.
dbname
 The name of your database, to which the data source connection refers.
schema

The name of your schema, to which the data source connection refers.
dialect

The dialect of the referenced database.

You can enter one of the following values:

  • azure, for an Azure SQL Server data source.
  • bigquery, for a Google BigQuery data source.
  • db2, for an IBM DB2 data source.
  • hana, for an SAP HANA data source.
  • hana-cviews, for getting lineage from calculated views in an SAP HANA Classic on-premises data source.
  • hana-cviews-v2, for getting lineage from calculated views in an SAP HANA Cloud/Advanced data source.
    Important 

    To get technical lineage including calculated views, you must harvest SAP HANA by specifying two data sources in the lineage harvester configuration file. In one data source, specify the hana dialect, and in the other, specify the hana-cviews or hana-cviews-v2 dialect.

  • hive, for a HiveQL data source.
  • greenplum, for a Greenplum data source.
  • mssql, for a Microsoft SQL Server data source.
  • mysql, for a MySQL data source.
  • netezza, for a Netezza data source.
  • oracle, for an Oracle data source.
  • postgres, for a PostgreSQL data source.
  • redshift, for an Amazon Redshift data source.
  • snowflake, for a Snowflake data source.
  • spark, for a Spark SQL data source.
  • sybase, for a Sybase data source.
  • teradata, for a Teradata data source.
collibraSystemName
 The system or server name of the data source.

Use this property with the useCollibraSystemName property in the lineage harvester configuration file to override the default Collibra System asset name for this data source.

Specify this property with the same name as the name of the System asset that you create when you prepare the physical data layer in Data Catalog. If you don't prepare the physical data layer, Collibra Data Lineage cannot stitch the data objects in your technical lineage to the assets in Data Catalog.

This property is optional.

Jobs

The jobs that you want the lineage harvester to collect and process to create the technical lineage.

This section is optional. The following rules apply when you specify this section:

  • Specify jobs that are executed so that the technical lineage graph does not include any job parameters with undefined values.
  • Specify only the first and parent jobs in a sequence of executed jobs. The lineage harvester automatically collects all jobs that are called by the parent jobs.

For details about how CollibraData Lineage parses DataStage jobs and resolves parameters, see Transformation logic and common errors for DataStage.
JobParameters
 The runtime parameters that are not in the DSX and ENV files. You can specify multiple job parameters.
name
 The name of the job parameter. You can specify any of the following values:
  • A parameter name
  • A user variable
  • A parameter set

Important Do not enclose the name between "#" characters, for example "name": "#name#"
value

The value of the job parameter. You can specify one of the following values, depending on the value of the name property:

  • If a parameter name is specified for the name property, specify one of the following values:
    • A parameter value
    • A parameter reference
  • If a user variable is specified for the name property, specify one of the following values:
    • A parameter value
    • A parameter set reference
  • If a parameter set is specified for the name property, specify this property with a value file name.

For details about how the values are resolved, see the "Parameter resolution" section in Transformation logic and common errors for DataStage.
perJobParameters

The parameters of a specific job. For example, you ingest multiple jobs where the parameters have the same name, but different values.

Note  This value takes precedence over the values specified in the JobParameters property. Otherwise, the original jobParameters field is used as the “default” option.
jobID
 The ID of the job.
name
 The name of the job parameter. You can specify any of the following values:
  • A parameter name
  • A user variable
  • A parameter set

Important Do not enclose the name between "#" characters, for example "name": "#name#"
value
The value of the job parameter. You can specify one of the following values, depending on the value of the name property:
  • If a parameter name is specified for the name property, specify one of the following values:
    • A parameter value
    • A parameter reference
  • If a user variable is specified for the name property, specify one of the following values:
    • A parameter value
    • A parameter set reference
  • If a parameter set is specified for the name property, specify a value file name as the value.

For details about how the values are resolved, see the "Parameter resolution" section in Transformation logic and common errors for DataStage.