Lineage harvester integrations available in beta

Collibra Data Intelligence Cloud supports many data sources and metadata sources, such as ETL tools or BI sources, for which you can create a technical lineage or which you can ingest.

Before Collibra releases a new lineage harvester, we test the new lineage harvester integrations extensively. However, we cannot foresee all possible use cases and scenarios. To further improve the lineage harvester, you can now test new lineage harvester integrations in beta. After a testing period, the new lineage harvester integrations become available for all Collibra Data Lineage users

Note Documentation is only available when the lineage harvester integrations are released. However, if you want to test new integrations, you can request testing guidelines and provide feedback.

The following table shows which integrations the lineage harvester currently supports in beta.

Metadata source	Available in lineage harvester version	Limitations	Beta process status
AWS Glue (script annotations)	1.4.0 and newer	The lineage harvester can process AWS Glue annotations in scripts coded in Python and Scala. Collibra Data Lineage does not stitch the AWS Glue metadata to Amazon S3 assets created by synchronizing an S3 File system or by registering a data source using the Collibra-provided AWS Glue driver.	Open
(Undefined variable: technical-lineage.AzureDataFactory)	2022.05.0-6 and newer	The result is a complete technical lineage that includes many Azure Data Factory transformations and some datasets. Flowlets are not yet supported. Transformations containing column patterns or rule-based mappings can only be partially analyzed, as they generate column names on the fly during the actual dataflow run. If technical lineage is detected from a dynamically generated column, it is given the placeholder Dynamic Column in the technical lineage viewer. Some reserved variables names, for example {@context}, are not yet supported.	Open

Warning The lineage harvester beta integrations offer early access to new integrations. However, we can only allow a limited number of customers to test the integrations and give feedback. We will make the integrations available for all customers after processing the feedback and improving the lineage harvester.

Testing an integration in beta

If you want to access the lineage harvester and the testing guidelines to test a lineage harvester integration in beta, do the following:

1. Create a support ticket to get access to the Technical lineage section of the Collibra Product Resources Downloads page and the testing guidelines for the lineage harvester integration.
   You now have access to the testing guidelines.
   You can now download and install the lineage harvester.
   Tip If you purchased Collibra Data Lineage you already have access to the newest harvester. However, you still have to create a support ticket to access the testing guidelines.
2. Test the lineage harvester integration in beta.
3. Reach out to Collibra to provide feedback via your CSM or a support ticket.