Supported Azure Data Factory transformation details

Collibra Data Lineage supports the most commonly used transformations, activities and datasets in Azure Data Factory (beta) .

Supported transformations

The following tables shows the non-exhaustive list of supported and unsupported transformations.

Supported transformations

Unsupported transformations
  • Aggregate1
  • Alter Row
  • Assert
  • Derived Column1
  • Exists
  • External Call2
  • Filter
  • Flatten1
  • Join
  • Lookup
  • Parse1
  • Pivot3
  • Rank
  • Select1
  • Sink4
  • Sort
  • Source
  • Split
  • Stringify
  • Surrogate Key
  • Union
  • Unpivot
  • Window1
  • Some reserved variables names, for example {@context}
  • Flowlets

Limitations

  1. Transformations that contain column patterns or rule-based mappings can only be partially analyzed because they generate column names on the fly during the actual data flow run. If technical lineage is detected from a dynamically generated column, it is given the placeholder Dynamic Column in the technical lineage viewer.
  2. In the Mapping section of the editor, column patterns are not supported and not displayed in the technical lineage graph. Note that Auto mapping uses column patterns behind the scenes and is therefore not supported either.
  3. Pivoted columns can only be inferred when explicit values are provided in the Pivot Key tab. When columns cannot be inferred, a placeholder Pivoted Columns is added.
  4. The SQL scripts and rule-based mappings in the transformation are not supported.

Supported datasets

The following table shows the non-exhaustive list of supported datasets:

Dataset

Definition
AmazonRedshiftTableDataset The Amazon Redshift table dataset.
AzureBlobDataset The Azure Blob storage.
AzureBlobFSDataset The Azure Data Lake Storage Gen2 storage.
AzureDataLakeStoreDataset The Azure Data Lake Store dataset.
AzureSqlDWTableDataset The Azure SQL Data Warehouse dataset.
AzureSqlMITableDataset The Azure SQL Managed Instance dataset.
AzureSqlTableDataset The Azure SQL Server database dataset.
CosmosDbSqlApiCollectionDataset Microsoft Azure CosmosDB (SQL API) Collection dataset.
Db2TableDataset The Db2 table dataset.
OdbcTableDataset The ODBC table dataset.
OracleTableDataset The on-premises Oracle database dataset.
SnowflakeDataset The snowflake dataset.
SqlServerTableDataset The on-premises SQL Server dataset.