Azure Data Factory: Supported transformation details

Collibra Data Lineage supports the most commonly used transformations and data sources in Azure Data Factory.

Pipelines

Technical lineage for Azure Data Factory retrieves and processes pipeline metadata and includes it in the technical lineage graph as follows:

If a pipeline definition does not contain parameters, Collibra Data Lineage processes the definition and includes the associated assets under the pipeline definition folder in the technical lineage tree in the technical lineage viewer:
- Pipeline Definition
  - Assets
If a pipeline definition contains parameters, Collibra Data Lineage processes pipeline runs and pipeline triggers to resolve parameter values. The assets are then included in the following structure in the technical lineage tree:
- Pipeline Definition
  - Pipeline Runs
    - Date
      - Pipeline run ID
        Assets
  - Pipelines Triggers
    - Pipeline name and run count
      - Assets

Note that only the definitions of pipeline triggers are processed and included in the technical lineage graph.

Collibra Data Lineage processes only parameter values that are explicitly provided during pipeline execution.
To exclude pipeline run and pipeline trigger metadata from technical lineage, set the Pipeline Runs Days To Look Back field to 0 in the technical lineage for ADF capability. In this case, only pipeline definitions are processed and included in the technical lineage graph. You can also use this field to specify how many days of pipeline run metadata to be collected and processed. For more information, go to Create a technical lineage via Edge for Azure Data Factory

Supported transformations

The following tables shows a non-exhaustive list of supported and unsupported transformations.

Supported transformations	Unsupported transformations
Aggregate¹ Alter Row Assert Derived Column¹ Exists External Call² Filter Flatten¹ Join Lookup Parse¹ Pivot³ Rank Select¹ Sink⁴ Sort Source Split Stringify Surrogate Key Union Unpivot Window¹	Some reserved variables names, for example {@context} Flowlets
Limitations Transformations that contain column patterns or rule-based mappings can only be partially analyzed because they generate column names on the fly during the actual data flow run. If technical lineage is detected from a dynamically generated column, it is given the placeholder Dynamic Column in the technical lineage viewer. In the Mapping section of the editor, column patterns are not supported and not displayed in the technical lineage graph. Note that Auto mapping uses column patterns behind the scenes and is therefore not supported either. Pivoted columns can only be inferred when explicit values are provided in the Pivot Key tab. When columns cannot be inferred, a placeholder Pivoted Columns is added. The SQL scripts and rule-based mappings in the transformation are not supported.

Supported transformations

Unsupported transformations

Aggregate¹
Alter Row
Assert
Derived Column¹
Exists
External Call²
Filter
Flatten¹
Join
Lookup
Parse¹
Pivot³
Rank
Select¹
Sink⁴
Sort
Source
Split
Stringify
Surrogate Key
Union
Unpivot
Window¹

Some reserved variables names, for example {@context}
Flowlets

Limitations

Transformations that contain column patterns or rule-based mappings can only be partially analyzed because they generate column names on the fly during the actual data flow run. If technical lineage is detected from a dynamically generated column, it is given the placeholder Dynamic Column in the technical lineage viewer.
In the Mapping section of the editor, column patterns are not supported and not displayed in the technical lineage graph. Note that Auto mapping uses column patterns behind the scenes and is therefore not supported either.
Pivoted columns can only be inferred when explicit values are provided in the Pivot Key tab. When columns cannot be inferred, a placeholder Pivoted Columns is added.
The SQL scripts and rule-based mappings in the transformation are not supported.

Supported data sources

The following table shows a non-exhaustive list of supported sources with the corresponding dataset and linked service types.

CollibraData Lineage supports all data format types that are supported in Azure Data Factory, including binary, Excel file, Delimited text, JSON, Parquet, and so on.

Data sources	Dataset type	Linked service type
Amazon Redshift	AmazonRedshiftTable	AmazonRedshift
Azure Blob storage	AzureBlob	AzureBlobStorage
Azure Data Lake Storage Gen2	AzureBlobFSFile	AzureBlobFS
Azure Data Lake Store	AzureDataLakeStoreFile	AzureDataLakeStore
Azure Databricks Delta Lake	AzureDatabricksDeltaLake	AzureDatabricksDeltaLake
Azure SQL Managed Instance	AzureSqlMITable	AzureSqlMI
Azure SQL Server database	AzureSqlTable	AzureSqlDatabase
Azure Synapse Analytics	AzureSqlDWTable	AzureSqlDW
DB2 data source	Db2Table	Db2
Google Cloud Storage	GoogleCloudStorageLocation	GoogleCloudStorage
Microsoft Access	MicrosoftAccessTable	MicrosoftAccess
Microsoft Azure Cosmos Database	CosmosDbSqlApiCollection	CosmosDb
Open Database Connectivity (ODBC)	OdbcTable	Odbc
On-premises Oracle database	OracleTable	Oracle
REST	RestResource	RestService
Salesforce	SalesforceObject	Salesforce
Salesforce Marketing Cloud	SalesforceMarketingCloudObject	SalesforceMarketingCloud
Salesforce Service Cloud	SalesforceServiceCloudObject	SalesforceServiceCloud
SAP Business Warehouse (open hub)	SapOpenHubTable	SapBW
SFTP server	SftpLocation	Sftp
Snowflake	SnowflakeTable	Snowflake
SQL Server	SqlServerTable	SqlServer

Supported activity types

A Data Factory can have one or more pipelines. A pipeline is a logical grouping of activities that together perform a task. There are three groupings of activities: data movement activities, data transformation activities, and control activities. For a complete list of Azure Data Factory activity types and descriptions, see Microsoft's documentation on pipelines and activities.

Collibra Data Lineage currently supports the following activity types:

Activity type	Activity group
Append Variable	Control flow
Copy	Data movement
Data Flow	Data transformation
Execute Pipeline	Control flow
For Each	Control flow
Get Metadata	Control flow
If Condition	Control flow
Lookup	Control flow
Set Variable	Control flow
Switch	Control flow
Until	Control flow
Web	Control flow