IBM InfoSphere DataStage uses jobs with stages instead of transformations. DataStage has three job types: parallel jobs, sequence jobs and server jobs. For a list of all job stages per job type in DataStage, read the IBM documentation.
Indirect lineage
Collibra Data Lineage supports indirect lineage for PxLookup stages. Indirect lineage is created for fields that are used in the Lookup transformation but are not part of the transformation output.
For more information about indirect lineage, go to The technical lineage graph and Technical lineage Settings tab pane.
Supported parameters and expressions
Technical lineage for DataStage supports the following parameters and expressions:
-
Runtime parameters in parameter set files.To include the runtime parameters, ensure to export DataStage files with executables. For more information about exporting DataStage files, go to Prepare an external directory folder for the lineage harvester (deprecated) if you use the lineage harvester (deprecated), or Create a technical lineage via Edge for DataStage.
- Parameter sets.To include parameters, export the parameter sets as part of your environment file. For more information about exporting DataStage files, go to Prepare an external directory folder for the lineage harvester (deprecated) if you use the lineage harvester (deprecated), or Create a technical lineage via Edge for DataStage.
- Expression format. The analysis result displays the DATASTAGE_EXPRESSION message when a complex format with advanced functions is parsed.
File patterns in PxSequentialFile stages
When a PxSequentialFile stage uses the filepattern property, the value typically contains wildcards, such as *, that DataStage resolves at runtime against the file system. Because the DataStage integration does not have access to the file system, it cannot determine the actual files that match the pattern. As a result, Collibra Data Lineage cannot create file assets or generate lineage for these file sources.
To generate file asset lineage, use the file property with a concrete path or a parameter value that can be resolved during analysis.
For details about how Collibra Data Lineage extracts lineage and resolves parameters from DataStage, go to Transformation logic and common errors for DataStage.