Depending on the integration, Collibra Data Lineage derives technical lineage from static analysis, runtime execution, or both. In this documentation, these approaches are referred to as design lineage and runtime lineage, respectively.
- Design lineage
- Design lineage is derived by statically analyzing code or pipeline definitions without executing them. It reflects the intended data flow: what your pipelines or scripts are designed to do, regardless of whether they have been executed.
Some design-lineage integrations support parameter resolution to resolve runtime values that are not available during static analysis. This improves lineage accuracy by resolving references such as table names, file paths, and connection details. If a parameter cannot be resolved, it appears as a placeholder in the technical lineage graph, and the lineage for that reference may be incomplete.
- Runtime lineage
- Runtime lineage is captured at runtime. It reflects the actual data flow, what your pipelines or scripts did when they ran, including the real values used during execution.
The following table shows examples of the lineage approach used by selected integrations. Unless otherwise noted in the integration documentation, supported integrations derive design lineage by statically analyzing code or pipeline definitions.
| Integration | Lineage approach | Description |
|---|---|---|
| DataStage | Design | Collibra Data Lineage statically analyzes job definitions. You can specify job parameter names and values in the source configuration to resolve parameters used in SQL statements and table references. For more information, go to Transformation logic and common errors for DataStage. |
| OpenLineage | Runtime | Lineage events are emitted by your data pipeline tools at runtime and captured by Collibra Data Lineage. |
| Azure Data Factory | Both | The ADF scanner captures pipeline definitions (design lineage) and pipeline run history (runtime lineage). |