Add the Sybase capability for Shared Storage connections

Add an Edge capability to authorize your Edge site to crawl the metadata and generate the technical lineage.

Important  Collibra Cloud sites do not support Shared Storage connections.

Required permissions

Steps

  1. Open a site.
    1. On the main toolbar, click Products icon → Cogwheel icon Settings.
      The Settings page opens.
    2. In the tab pane, click Edge.
      The Sites tab opens and shows a table with an overview of your sites.
    3. In the table, click the name of the site whose status is Healthy.
      The site page opens.
  2. In the Capabilities section, click Add capability.
    The Add capability page appears.
  3. Select the Technical Lineage for SqlDirectory capability template.
  4. Enter the required information.
    FieldDescriptionRequired?

    Name

    The name of the capability.

    Yes

    Description

    The description of the capability.

    Yes

    Capability template

    The capability template. The value that you select in this field determines which sections appear on the page.

    Select the following capability:

    Technical Lineage for SqlDirectory

    Yes

    Source ID

    The name of the data source. The name must be unique and cannot contain special characters, for example, /.

    Yes

    TechLin Admin Connection (in preview)

    If you want to use the OAuth authentication type to connect to the Collibra Data Lineage service instances, you have to create a Technical Lineage Admin Edge or Collibra Cloud site connection and select the OAuth authentication type. Then, in this field, you specify the name of the Technical Lineage Admin Edge or Collibra Cloud site connection.

    No

    Shared Storage Connection

    The Shared Storage connection that you created.

    No

    Mask

    The pattern of the file names in the directory. By default, the value is *.

    Yes

    Dialect

    The dialect of the database.

    You can enter one of the following values:

    • azure, for an Azure SQL Server data source.
    • bigquery, for a Google BigQuery data source.
    • db2, for an IBM DB2 data source.
    • hana, for an SAP HANA data source.
    • hana-cviews, for getting lineage from calculated views in an SAP HANA Classic on-premises data source.
    • hana-cviews-v2, for getting lineage from calculated views in an SAP HANA Cloud/Advanced data source.
      Important 

      To get technical lineage including calculated views, you must harvest SAP HANA by adding two Technical Lineage for SqlDirectory capabilities with the Shared Storage connections. In one capability, specify the hana dialect, and in the other, specify the hana-cviews or hana-cviews-v2 dialect.

    • hive, for a HiveQL data source.
    • greenplum, for a Greenplum data source.
    • mssql, for a Microsoft SQL Server data source.
    • mysql, for a MySQL data source.
    • netezza, for a Netezza data source.
    • oracle, for an Oracle data source.
    • postgres, for a PostgreSQL data source.
    • redshift, for an Amazon Redshift data source.
    • snowflake, for a Snowflake data source.
    • spark, for a Spark SQL data source.
    • sybase, for a Sybase data source.
    • teradata, for a Teradata data source.

    Yes

    Collibra System Name

    The system or server name of the data source. This field is also the full name of your System asset in Data Catalog.

    The value of this field must be the same as the full name of the System asset that you created when you registered the data source.

    Yes

    Database

    The name of your database, which is also the name of your Database asset in Data Catalog.

    Note The database and schema names in the SQL statements in your SQL files take precedence over the values that you provide for the Database and Schema fields in the technical lineage for SqlDirectory capability. If your SQL statements contain database and schema names, Collibra Data Lineage uses them for stitching. If your SQL statements do not contain database and schema names, Collibra Data Lineage uses the values of the Database and Schema fields in the capability for stitching. Fore more information, go to Prepare the SQL directory and Automatic stitching for technical lineage.

    Yes

    Schema

    The name of the default schema, if not specified in the data source itself. This corresponds to the name of your Schema asset.

    Note The database and schema names in the SQL statements in your SQL files take precedence over the values that you provide for the Database and Schema fields in the technical lineage for SqlDirectory capability. If your SQL statements contain database and schema names, Collibra Data Lineage uses them for stitching. If your SQL statements do not contain database and schema names, Collibra Data Lineage uses the values of the Database and Schema fields in the capability for stitching. Fore more information, go to Prepare the SQL directory and Automatic stitching for technical lineage.

    Yes

    Database-System mapping

    This optional field allows you to map databases to their rightful systems, to obtain stitching. This resolves missing stitching, which occurs when Collibra Data Lineage associates multiple databases with the default system name that you provide in the Collibra System Name field.

    Let’s say you use the Collibra System Name field to specify the system name SystemA. This system name will be associated with all of the databases mentioned in your SQL statements in your SQL files.

    Now let’s say that you have a SQL statement that selects data from a database DB2 (which is in SystemB) and inserts the data in database DB3 (which is part of SystemC). Both of these databases will be associated with SystemA, and both will appear under SystemA in the Browse tab pane. Therefore, stitching fails and duplicate databases might appear.

    To resolve this issue and obtain stitching, you can use this field to map DB2 to its rightful SystemB, and DB3 to its rightful SystemC.

    No

    Property

    This section contains the custom parameters you can specify to create technical lineage. Click Add property to add a property.

    You can use this field to set the HTTP timeout duration by adding the httpTimeout property: 

    TypeValue TypeNameExample value

    Text

    Plaintext

    httpTimeout

    Sets the HTTP timeout duration, in seconds. You can enter a value in the range of 1 to 3599. The default value is 15.

    		15
    Warning If you are a Collibra Platform for Government customer, this field is required to connect to a Collibra Data Lineage service instance:
    TypeValue TypeNameValue

    Text

    Plaintext

    techlinHost

    This is the URL of the Collibra Data Lineage service instance to which you want to upload metadata, for example techlin-europe-west1.collibra.com.

    		Example: techlin-europe-west1.collibra.com

    Text

    Secret

    techlinKey

    This is the unique API key to connect to a Collibra Data Lineage service instance.

    Specify a unique user key for each Collibra environment. If you're not sure what your user key is, contact your Collibra Collibra Account Team.

    		<your-techlin-key>

    Yes for US government customers.

    Dependent On Sources

    This option allows you to provide table-definition details from an independent data source to a data source that is dependent on those details. This is needed to avoid analysis errors and to have a complete lineage that includes lineage from the SQL statements from dependent data sources.

    To use this option, enter the source ID of the independent source.

    Let's consider an example. Let's say that you want to create a technical lineage for two data sources:

    Database1 contains the DDL that specifies that the database has a table named "Table1", which has three columns: Col1, Col2, Col3, and Col4.

    Database2 contains an SQL statement: SELECT * from Database1.Schema1.Table1.

    The SQL statement in Database2 refers to the table in Database1. Therefore, to get lineage from the statement in Database2, the table definition from Database1 must be known. In this case, we say that Database2 is dependent on Database1. Database1 is considered the independent data source.

    To configure this option, specify the Source ID of the independent data source, in this example, Database1, as shown in the following imagee

    An image showing the Dependent on Sources field in the Edge capability

    Important If a dependent data source contains lowercase column names, this feature will only work for the following dialects: Oracle, Snowflake, and Teradata. For all other dialects:
    • An analyze error is raised, prompting you to provide the DDL file.
    • The only workaround is to consolidate your SQL statements and DDL file in a single data source.

    For complete information, go to Sharing database models across data sources.

    		 
    Database Link Mapping

    If you are using DBLinks, this optional field allows you to configure, per data source, the database and schema to which DBLink points.

    The configuration format is as follows:

    {"<dblink_name>": {"database":"<database>","schema":"<schema>"}, ...}

    The schema provided here is only taken into consideration if a schema is not explicitly specified in the SQL query. As such, the schema specified here can be considered a default or fallback mapping.

    Tip  If you’re using a DBLink to target another source, you need to share the databasae model between the targeted (independent) source and the dependent source. Use the Dependent On Sources option to configure that dependency and share the database model.

    Let's say that you have two Oracle data sources, A and B.

    Source A

    You have the following database configuration:

    • Database_A > Schema_A1 > Table_1 > Column_1, Column_2
    • Database_A > Schema_A2 > Table_2 > Column_x, Column_y

    Source B

    You have DBLinks with names dblink.example.com, dblink2.example.com, and dblink3.example.com, all of which point to Source A.

    Tip Configure the Dependent On Source option, recognizing the fact that Source B is dependent on target (independent) Source A.

    You want your DBLink aliases configured as follows:

    • dblink.example.com → Database_A > Schema_A1
    • dblink2.example.com → Database_A > Schema_A1
    • dblink3.example.com → Database_A > Schema_A2

    You specify this in the Database Link Mapping field as follows:

    {
	"dblink.example.com": {"database":"Database_A","schema":"Schema_A1"},
	"dblink2.example.com": {"database":"Database_A","schema":"Schema_A1"},
	"dblink3.example.com": {"database":"Database_A","schema":"Schema_A2"}
	}

    With this done, the following example queries on Source B are successfully analyzed by the Collibra Data Lineage service instance:

    Multiple dblinks under the same database scope

    You can configure multiple dblinks under the same database scope:

    {
	"dbScope1": {
		"dblink.example.com": {
			"database": "Database_A",
			"schema": "Schema_A1"
		},
		"dblink2.example.com": {
			"database": "Database_A",
			"schema": "Schema_A1"
		},
		"dblink3.example.com": {
			"database": "Database_A",
			"schema": "Schema_A1"
		}
	}	
}

    Multiple dblinks under the same schema scope

    You can configure multiple dblinks under the same schema scope. In the following example, DB1 is scoped for the entire database. DB2 is scoped based on the schema.

    {
    "DB1": {
        "[DB LINK NAME]": {
            "database": "[REMOTE DATABASE NAME]",
            "schema": "[REMOTE DATABASE DEFAULT SCHEMA]"
        }
    },
    "DB2": {
        "SCHEMA1": {
            "[DB LINK NAME]": {
                "database": "[REMOTE DATABASE NAME]",
                "schema": "[REMOTE DATABASE DEFAULT SCHEMA]"
            }
        },
        "SCHEMA2": {
            "[DB LINK NAME]": {
                "database": "[REMOTE DATABASE NAME]",
                "schema": "[REMOTE DATABASE DEFAULT SCHEMA]"
            }
        }
    }
}

    Important If the same DBLink, for example dblink.example.com, exists in multiple databases, the formatting shown in the previous example still applies, but you need to enclose it in curly brackets and specify the relevant database, as follows:
    • Basic formatting, as shown in the previous example:
      "dblink.example.com": {"database":"Database_A","schema":"Schema_A1"}
    • Formatting if the DBLink exists in multiple databases and you want to apply it only in a database named "dbScope1":
      "dbScope1": {"dblink.example.com": {"database":"Database_A","schema":"Schema_A1"}}

    If a DBLink is referenced in multiple mappings, as shown in the following example, the first mapping is used.

    "dbScope1": {
   "dblink.example.com": {"database":"DevDB_A","schema":"DevSch_A1"}
}, 
   "dblink.example.com": {"database":"Database_A","schema":"Schema_A1"}}

    In this case, occurrences of dblink.example.com in the database named "dbScope1" are mapped to:

    "database":"DevDB_A","schema":"DevSch_A1"

    No

    Delete Raw Metadata After Processing

    Technical lineage via Edge harvests raw metadata from specified data sources and uploads it in a ZIP file to a Collibra Data Lineage service instance. This option indicates whether the raw metadata should be deleted from the Collibra Data Lineage service instance after the metadata that is targeted for ingestion in Data Catalog is processed.

    Select this option to indicate that the raw source metadata is deleted after processing.

    Clear the checkbox to keep the raw source metadata after processing. In this case, it is stored in the Collibra infrastructure.

    No

    Analyze Only (Deprecated)

    Important This option is deprecated and will be removed in a future version of Collibra. We recommend that you no longer use it. The mandatory Processing Level setting, below, replaces this option.
    • The "Analyze" option in the Processing Level setting is the equivalent of selecting the Analyze Only option.
    • The "Sync" option in the Processing Level setting is the equivalent of clearing the Analyze Only option.

    No

    Processing Level

    Important This setting replaces the deprecated Analyze Only option, which will be removed in a future version of Collibra.

    For each of your data sources, you have to specify one of the following values: Load, Analyze, or Sync. Then, when you synchronize your technical lineage, the following process begins:

    1. Metadata for all data sources is loaded, regardless of the value of this setting for a particular data source.
    2. Metadata from data sources for which the value of this setting is either Analyze or Sync, is analyzed.
    3. Metadata from data sources for which the value of this setting is Sync, is synchronized.

    ValueDescription
    Load

    Harvest metadata from the data source and upload it to your Collibra environment. This allows you to inspect and, if necessary, edit the harvested metadata before uploading it to the Collibra Data Lineage service instance for analysis.

    When the job is done, you can download and review the metadata:

    1. Open the Activities list.
    2. In the row containing the job, click Result.
      The Synchronization Results dialog box appears.
    3. Click download and save the ZIP file to your hard drive.

    Tip The download link resembles the following: https://integrations.collibra-abc.com/rest/2.0/files/01944f12-7665-7d9c-8bc5-aa426b6a63cc. Take note of the file ID, in this example: 01944f12-7665-7d9c-8bc5-aa426b6a63cc. After you inspect the metadata, you can send the ZIP file for analysis by using the "Analyze files" option. Alternatively, you can upload the ZIP file using the POST /files API. In either case, you need to specify the file ID.

    Analyze

    Load and analyze the metadata on the Collibra Data Lineage service instance.

    Synchronization does not start after analysis; it starts only after either:

    Important  If you want to synchronize multiple data sources, we strongly recommend that you select this option in the respective Edge or Collibra Cloud site capabilities for each of your data sources. This allows you to synchronize all data sources in a single job, thereby maximizing efficiency and mitigating the risk of failed synchronization jobs.
    Sync

    Load, analyze, and synchronize metadata from all data sources. Synchronization starts – or is queued, if another synchronization job is running – immediately after analysis.

    Important If you want to synchronize multiple data sources and you select this option, each data source is processed as a separate job. This is highly inefficient and will likely lead to failed sync jobs. For complete information and important considerations, go to Tips for successful lineage synchronization.

    Yes

    Active

    The option determines whether to include or remove the technical lineage of the data source.

    Select this option to include the technical lineage of this data source.

    Clear the checkbox to exclude the technical lineage of this data source.

    Yes

  5. Click Create.
    The capability is added to the Edge or Collibra Cloud site.
    The fields become read-only.

What's next

You can now synchronize the technical lineage.