dbt Cloud source configuration

Updated:

Note This topic is only relevant if you are creating technical lineage via Edge. If you are using the CLI lineage harvester (deprecated), you need to create a <source ID> configuration file. The CLI harvester will officially reach its End of Life on July 31, 2026.

The Source configuration field in the dbt Cloud technical lineage Edge capability allows you to reduce the amount of data objects to be downloaded and enhance the lineage harvester performance in the following ways:

The value of the Source configuration field must be a valid block of JSON code, for example:

Copy 
{
    "collibraSystemNames":{
        "projects":[
        {
            "project_id":654321,
            "collibraSystemName":"SystemName"
        }
       ]
    },
    "filter":{
        "jobIds":[1234],
        "projectIds":[654321]
    },
    "materializedMapping":{
        "ELS_MATERIALIZE_MULTIPLE_EXTERNAL_TABLES":"VIEW"
    }
}

The following table describes the various properties you can use in your JSON code block.

Property

 Description

Required?
collibraSystemNames

You can use this section to specify the Collibra System Name for each project.

No
projects

This section contains the project names and the Collibra system names.

No
project_id

Your project ID. You can find the project ID in the dbt URL right after projects. For example, if your dbt URL is https://cloud.getdbt.com/develop/54321/projects/12345 , your project_id is 12345.

No
collibraSystemName

The system or server name of the data source. This is also the name of your System asset in Data Catalog:

Specify this property with the same name as the name of the System asset that you created when you registered the data source.

In the following code example, the project with the 12345 project ID is stitched to the systemname1 System asset in Data Catalog. 

{
   "collibraSystemNames":{
	"projects":[
	    {"project_id":"12345","collibraSystemName":"systemname1"}
	]
   },
}
No
filter

You can use this section to include projects and jobs to be downloaded. Collibra Data Lineage downloads and processes only the specified jobs and projects.

In the following code example, the job with the 1234 job ID and the projects with the 98 and 5678 project IDs are downloaded. 

{
       "filter": {
	   "jobIds": [
	       1234
	   ],
	   "projectIds": [
	       98,
	       5678
	   ]
	}
}

No
jobIds

The job IDs of the jobs that you want to include.

Specify an integer. Do not specify a string.

To get your job ID, in your dbt, select Deploy and then Jobs. Select a job and you can find your job ID in the URL. For example, if your URL is cloud.getdbt.com/deploy/65432/projects/23456/jobs/123456, 123456 is your job ID.

No
projectIds

The project IDs of the projects that you want to include.

Specify an integer. Do not specify a string.

You can find the project ID in the dbt URL right after projects. For example, if your dbt URL is https://cloud.getdbt.com/develop/54321/projects/12345 , your project_id is 12345.

 No
materializedMapping

Indicates how materializations in dbt are mapped. If you do not specify this property, CollibraData Lineage maps materializations to tables by default. You can change the mapping of a materialization to view.

In the following example, the ELS_MATERIALIZE_MULTIPLE_EXTERNAL_TABLES materialization is mapped to a view. 

"materializedMapping":{
    "ELS_MATERIALIZE_MULTIPLE_EXTERNAL_TABLES":"VIEW"
}
 No