Technical lineage Sources tab page

When you create a technical lineage, your data sources are uploaded to the Collibra Data Lineage service to be analyzed and processed. The Sources tab page shows the transformation details or source code that was analyzed and the results of this analysis.

You can access the Sources tab page by clicking Show status on the Settings tab pane.

Note If an analyzed data source has the following result, the data source does not appear in the Sources tab page:
Parsing errors: 0

Analysis errors: 0

Done: 0

No

 Name Description

Summary per data source

A summary per data source. You can also select data sources to filter the results.
Selected

Checkboxes to filter on a data source in the transformations table. If you select none, the transformations table contains all transformations.
Source ID

The ID of your data source. You entered this ID in the configuration file.
Scanner type

The type of scanner that is used to scan the queries in your data source.
Success rate

The success rate of the data source analysis on the Collibra Data Lineage service. The success rate indicates how complete your technical lineage is.

Important The success rate of a technical lineage gives a good indication of the processing success. A success rate less than 100%, however, does not mean processing was unsuccessful. A parsing error, for example, which negatively affects the success rate, does not always negatively affect the completeness of the lineage.
Done

The amount of queries that were scanned and analyzed.
Parsing Error

The amount of parsing errors.
Analyze Error

The amount of analysis errors.
Last sync time
 The last time the data source was uploaded to the Collibra Data Lineage service, for analysis and processing.

Search tools Tools to help you search for specific source code fragments.
Full-text search

A search field to find specific queries in the log files. Type what you are looking for and press Enter.

Filter by

A drop-down list to filter the source codes based on their status code.

Transformations table

The table that contains details of the transformations and source code (fragments).

You can filter the rows in the table by selecting data sources in the data source table and by using the search tools.

Tip If you click a source code fragment, you can see the log file attached to it.

Note Unity Catalog does not provide source code for each transformation. Therefore, when you create technical lineage for Databricks Unity Catalog, the source code pane does not display any source code.
ID
 The ID of the source code fragments or transformation details, which are assigned in chronological order.
Name

The name of the specific source code fragment or transformation detail.

You can also see the source code fragment name in the source code pane in the technical lineage graph.
Status code

The status of the analysis.

A source code fragment or transformation detail can have one of the following status codes:

  • DONE: All queries are processed successfully.
  • ERROR: Some queries could not be processed.
  • PARSING_ERROR: The syntax of some queries is invalid or unidentified.
  • ANALYZE_ERROR: Some columns are not linked to a table.
Status description

The description of the status code that provides more information about the analysis and shows how many queries were processed.
Group name

The name of the package or procedure to which the source code fragment or transformation details belongs.
Export Selected Transformations The button to export transformation details for the selected data sources. When you click this button, you download a ZIP file. The ZIP file contains an errors.csv file that includes the transformation details for the selected data sources. If you do not select any data sources, the transformation details for all listed data sources in the transformations table are exported.

Show lineage

The button to go back to the technical lineage graph.
Sort by each column The sorting icons that you can use to sort by each column in ascending or descending order. These columns include Scanner type, Success rate, Done, Parsing Error, Analyze Error, and Last sync time.

Analysis results

If you click one of the rows in the Transformations table, a file with the analysis results attached to the source code or transformation details opens. You can use these files to easily find errors in the source code or transformation details of your data source.

If the metadata that Collibra Data Lineage collects from your data source includes SQL queries, the analysis results might display comments from those SQL queries.

Note If a comment ends with a statement separator, for example, /*select 2 from dual*/;, the comment is counted as a statement. Consequently, the number of queries that are displayed in the Done column under Summary per data source might be greater than the actual number of queries parsed.

Analyze issues and possible solutions

If the analysis results contain errors, the Status code column shows ANALYZE_ERROR and the Name column displays the name of the error. You can find the transformation details by clicking the row.

The following list shows some common issues and errors, along with possible solutions:

When you review the analysis results of DataStage, you might encounter a DATASTAGE_EXPRESSION substitution even when there is no error.

Collibra Data Lineage can parse and create technical lineage for complex formats with advanced functions and statements, such as Trim, Field, and If...Then...Else. When such a complex expression is parsed, it is substituted with the string "DATASTAGE_EXPRESSION".

No action is required for this message.

For more information, go to Expression format in IBM InfoSphere Information Server documentation.

This error occurs when Collibra Data Lineage cannot find the parameter files that are listed in the error message. To resolve this issue:

  1. Ensure you have the parameter files.
  2. Take one of the following actions: 

Copy code

This error occurs when the taskflow type was specified in the objects to be retrieved, but Collibra Data Lineage could not find any taskflows while creating technical lineage for IICS. To resolve this issue, take any of the following actions:

Note that the only meaningful types to retrieve are: Taskflow, Workflow, Project and Folder.

Copy code

This error occurs in the following scenarios:

The unspecified properties are marked as UNDEFINED in the transformation details. To resolve this issue, specify the properties.

Copy code

This error occurs when parameters are missing from the parameter files in the external folder. To resolve this issue:

Copy code

This error occurs in any of the following scenarios:

The connections lacking definitions are listed when you expand the CONFIGURATION analyze error. To resolve this issue, take any of the following actions:

  • If you use technical lineage via Edge to create technical lineage,
    1. Copy the connection definitions that are marked as UNDEFINED in the error.
    2. Add the copied connection definitions to the Source Configuration field in the Technical Lineage for SQL Server Integration Services capability.
    3. Replace UNDEFINED with the correct values:
      • Ensure that the values for collibraSystemName, schema, and dbname match the system, schema, and database names in Data Catalog.
      • Select the value for dialect from the dialect list that Collibra Data Lineage supports.
      For more information about specifying the values, go to the description of the Source Configuration field in the Add a technical lineage capability to your Edge site step.
  • If you use the lineage harvester to create technical lineage,
    1. Copy the connection definitions that are marked as UNDEFINED in the error.
    2. Add the copied connection definitions to the connection_definitions.conf file .
    3. Replace UNDEFINED with the correct values:
      • Ensure that the values for collibraSystemName, schema, and dbname match the system, schema, and database names in Data Catalog.
      • Select the value for dialect from the dialect list that Collibra Data Lineage supports.
      For more information about specifying the values, go to the Configure the connection definitions step in Prepare an external directory folder for the lineage harvester.