Add the MicroStrategy technical lineage capability

After you have created a connection to MicroStrategy in your Edge or Collibra Cloud site, you have to add the MicroStrategy technical lineage capability to the connection.

Required permissions

Steps

  1. Open a site.
    1. On the main toolbar, click Products icon → Cogwheel icon Settings.
      The Settings page opens.
    2. In the tab pane, click Edge.
      The Sites tab opens and shows a table with an overview of your sites.
    3. In the table, click the name of the site whose status is Healthy.
      The site page opens.
  2. In the Capabilities section, click Add capability.
    The Add capability page appears.
  3. Select the Technical Lineage for MicroStrategy capability template.
  4. Enter the required information.
    FieldDescriptionRequired

    Name

    The name of the capability.

    Yes

    Description

    The description of the capability.

    No

    Source ID

    The name of the data source. The name must be unique and cannot contain special characters, for example, /.

    Warning 
    • You can only specify one source ID per MicroStrategy Intelligence Server. Ingesting the same MicroStrategy Intelligence Server under different source IDs will fail.
    • Any single MicroStrategy Intelligence Server can be ingested only once. If you create more than one connection for the same MicroStrategy Intelligence Server, integration will fail. If you want to ingest from multiple unique MicroStrategy Intelligence Servers, you have to create a new Edge connection for each one, configure a new capability template for each one, and each must have a unique source ID.

    Warning If you are switching between the lineage harvester and Edge, the value in this field must exactly match the value of the id property in your lineage harvester configuration file.

    We highly recommend that you specify only one source ID per MicroStrategy Intelligence Server.

    Yes

    TechLin Admin Connection (in preview)

    If you want to use the OAuth authentication type to connect to the Collibra Data Lineage service instances, you have to create a Technical Lineage Admin Edge or Collibra Cloud site connection and select the OAuth authentication type. Then, in this field, you specify the name of the Technical Lineage Admin Edge or Collibra Cloud site connection.

    No

    MicroStrategy connection

    The MicroStrategy connection that you created for ingestion in Data Catalog.

    Tip Select the name that you provided in the Name field when you created a connection to MicroStrategy.

    Yes

    Domain ID

    The unique reference ID of the domain in Collibra Platform in which you want to ingest the MicroStrategy assets.

    Open the relevant domain in Collibra. The URL looks like: https://<yourcollibrainstance>/domain/22258f64-40b6-4b16-9c08-c95f8ec0da26?view=00000000-0000-0000-0000-000000040001. In this example, the reference ID is in bold.

    		 Yes

    URL for reports

    		This optional property ensures that the correct URL to data objects in MicroStrategy is included on the asset pages of corresponding MicroStrategy assets. The required value depends on which platform you run MicroStrategy:
    • For J2EE, use: "MicroStrategy/servlet/mstrWeb"
    • For .NET, use: "MicroStrategy/asp/Main.aspx"

    No

    MicroStrategy Library URL

    If you are using a custom URL to connect to the MicroStrategy Library Server, use this field to specify the custom library URL.

    Important You only need to specify the URL if both of the following are true:
    • You are connecting to a proxy server.
    • You are not using the default, hardcoded URL to the MicroStrategy Library Server.

      Example If the URL to your MicroStrategy Library is https://collibra.microstrategy.com/MicroStrategyLibrary/api, you don't need to use this field, as that is the default, hardcoded URL. However, if the URL is something like https://collibra.microstrategy.com/MicroStrategyLibraryProd/api, then use this field and configure it as follows:
      "microStrategyLibraryUrl": "MicroStrategyLibraryProd"

    		 No

    Source configuration

    This field allows you to provide JSON code, to:

    • Specify the default domain, meaning the domain in Collibra in which the corresponding assets of MicroStrategy metadata will be ingested if domain mapping is not configured.
      If you do configure domain mapping, the default domain is still the destination domain of the MicroStrategy Server asset.
    • Optionally, specify from which MicroStrategy projects you want to ingest metadata, and into which domains you want to ingest the corresponding assets.
    • Optionally, configure data source mapping, to map the name of a data source returned by the lineage harvester to the true name of the data source.
      Note Mapping doesn't work for custom SQL.

    If you previously integrated MicroStrategy via the lineage harvester, you can copy and paste in this field the JSON code from your MicroStrategy<source ID> configuration file.

    Property

    Description

    Mandatory

    default_domain_id

    The domain in which you want the corresponding assets of MicroStrategy metadata to be ingested.

    If you configure filtering, only the MicroStrategy Server asset is ingested into this default domain.

    Yes

    filters

    This section allows you to specify:

    • From which MicroStrategy projects you want to harvest metadata.
    • Into which domains in Collibra you want to ingest the corresponding assets.

    If you don't want to filter on projects, don't include this section in your <source ID> configuration file.

    No

    domainId

    The unique resource ID of the domain (or domains) in Collibra in which you want to ingest the MicroStrategy assets.

    Tip If you use a filters section, you must include the domainId property in the section. If, by chance, you want to filter on certain projects, but you want to ingest all assets into the default domain, then the value of the domainId property must match the value of the default_domain_id property.

    For example:
    "default_domain_id": "1234567890",
    "filters": [
        {
            "domainId": "1234567890",
            "projectNames": ["MicroStrategy Tutorial","Testing_MSTR"]
        },

    To find a domain reference ID, open the relevant domain in Collibra. The URL looks like: https://<yourcollibrainstance>/domain/22258f64-40b6-4b16-9c08-c95f8ec0da26?view=00000000-0000-0000-0000-000000040001. In this example, the reference ID is in bold.

    Yes

    projectIds
    		The IDs of the MicroStrategy projects from which you want to ingest metadata.

    No

    projectNames
    		The project names of the MicroStrategy projects from which you want to ingest metadata.

    No

    datasourceMapping

    This optional section allows you to configure data source mapping. Include this section only if you need to differentiate between multiple data sources that have the same name.

    Note Mapping doesn't work for custom SQL.

    No

    found_datasource

    The name of the data source that was returned by the lineage harvester, as shown in the technical lineage.

    Note The data source name is case-sensitive.

    Yes

    found_project

    The name of the project in which the data source information resides. You can specify an asterisk (*) to search for data source information across all projects.

    Yes

    mapping

    Use this section to map the data source name that was returned by the lineage harvester to the true name of the data source.

    Example You have a Redshift data source named "RD_pearl", but the lineage harvester has returned the name "Redshift_connection". You can configure the datasourceMapping section as follows:
    Copy
    {
        "datasourceMapping": [
         {
             "found_datasource": "REDSHIFT",
             "found_project": "*",
             "mapping": {
              "dbname": "RD_pearl",
              "collibraSystemName": "TV_dev"
             }
         }
        ]
    }

    Yes

    dbname

    The name of the database to which you want to map the found data source.

    Yes

    schema_name

    The name of the schema in MicroStrategy.

    No

    dialect

    The dialect of the data source in MicroStrategy.

    No

    collibraSystemName

    The system or server name of a database.

    Warning The values of this property must exactly match the name of your System asset in Collibra.

    If you set the useCollibraSystemName property to true in your lineage harvester configuration file, but you either don't create a <source ID> configuration file, or don't specify a value for the collibraSystemName property in your <source ID> configuration file, the system name in the technical lineage is "DEFAULT".

    If you set the useCollibraSystemName property to false in your lineage harvester configuration file, leave this property empty as follows: "collibraSystemName": "".

    How to configure this property if you have two databases with the same name

    Let's assume that you have a data source named Customers. You use this data source connection in two different projects, Project_A and Project_B, but they are actually two different databases. When you prepare the physical data layer in Data Catalog, you create a System asset for each of these databases. Let's say you named them Customers-North and Customers-South. You can then configure this property as follows.

    Copy
    "datasourceMapping": [
            {
                "found_datasource": "Customers",
                "found_project": "Project_A",
                "mapping": {
                    "dbname": "Customers",
                    "collibraSystemName": "Customers_North"
                }
            },
            {
                "found_datasource": "Customers",
                "found_project": "Project_B",
                "mapping": {
                    "dbname": "Customers",
                    "collibraSystemName": "Customers_South"
                }
            }
        ]

    Yes

    Example 
    .
    Copy
    {
        "default_domain_id": "66ab7911-d84a-43d6-b9f7-40130jd84ds98",
        "filters": [
            {
                "domainId": "3ae946ec-9030-46b8-ad86-a217b8f99bab",
                "projectIDs": ["3FF7CF7D4C5A2AEBF14D8B9802698B27"]
            },
            {
                "domainId": "21cfb3ae-0616-478e-9060-114130540be5",
                "projectNames": ["Human Resources Analysis Module"]
            }
        ],
        "datasourceMapping": [
         {
             "found_datasource": "REDSHIFT",
             "found_project": "*",
             "mapping": {
                 "dbname": "RD_pearl",
                  "schema_name": "Default_North",
                  "dialect": "spark",
                  "collibraSystemName": "TV_dev"
             }
         }
        ]
    }
    		 No
    Property

    This section contains the custom parameters you can specify to create technical lineage. Click Add property to add a property.

    You can use this field to set the HTTP timeout duration by adding the httpTimeout property: 

    TypeValue TypeNameExample value

    Text

    Plaintext

    httpTimeout

    Sets the HTTP timeout duration, in seconds. You can enter a value in the range of 1 to 3599. The default value is 15.

    		15
    Warning If you are a Collibra Platform for Government customer, this field is required to connect to a Collibra Data Lineage service instance:
    TypeValue TypeNameValue

    Text

    Plaintext

    techlinHost

    This is the URL of the Collibra Data Lineage service instance to which you want to upload metadata, for example techlin-europe-west1.collibra.com.

    		Example: techlin-europe-west1.collibra.com

    Text

    Secret

    techlinKey

    This is the unique API key to connect to a Collibra Data Lineage service instance.

    Specify a unique user key for each Collibra environment. If you're not sure what your user key is, contact your Collibra Collibra Account Team.

    		<your-techlin-key>

    Yes for US government customers.

    Dependent On Sources

    This option allows you to provide table-definition details from an independent data source to a data source that is dependent on those details. This is needed to avoid analysis errors and to have a complete lineage that includes lineage from the SQL statements from dependent data sources.

    To use this option, enter the source ID of the independent source.

    Let's consider an example. Let's say that you want to create a technical lineage for two data sources:

    Database1 contains the DDL that specifies that the database has a table named "Table1", which has three columns: Col1, Col2, Col3, and Col4.

    Database2 contains an SQL statement: SELECT * from Database1.Schema1.Table1.

    The SQL statement in Database2 refers to the table in Database1. Therefore, to get lineage from the statement in Database2, the table definition from Database1 must be known. In this case, we say that Database2 is dependent on Database1. Database1 is considered the independent data source.

    To configure this option, specify the Source ID of the independent data source, in this example, Database1, as shown in the following imagee

    An image showing the Dependent on Sources field in the Edge capability

    Important If a dependent data source contains lowercase column names, this feature will only work for the following dialects: Oracle, Snowflake, and Teradata. For all other dialects:
    • An analyze error is raised, prompting you to provide the DDL file.
    • The only workaround is to consolidate your SQL statements and DDL file in a single data source.

    For complete information, go to Sharing database models across data sources.

    No

    Delete Raw Metadata After Processing

    Technical lineage via Edge harvests raw metadata from specified data sources and uploads it in a ZIP file to a Collibra Data Lineage service instance. This option indicates whether the raw metadata should be deleted from the Collibra Data Lineage service instance after the metadata that is targeted for ingestion in Data Catalog is processed.

    Select this option to indicate that the raw source metadata is deleted after processing.

    Clear the checkbox to keep the raw source metadata after processing. In this case, it is stored in the Collibra infrastructure.

    		 No

    Analyze Only (Deprecated)

    Important This option is deprecated and will be removed in a future version of Collibra. We recommend that you no longer use it. The mandatory Processing Level setting, below, replaces this option.
    • The "Analyze" option in the Processing Level setting is the equivalent of selecting the Analyze Only option.
    • The "Sync" option in the Processing Level setting is the equivalent of clearing the Analyze Only option.

    No

    Processing Level

    Important This setting replaces the deprecated Analyze Only option, which will be removed in a future version of Collibra.

    For each of your data sources, you have to specify one of the following values: Load, Analyze, or Sync. Then, when you synchronize your technical lineage, the following process begins:

    1. Metadata for all data sources is loaded, regardless of the value of this setting for a particular data source.
    2. Metadata from data sources for which the value of this setting is either Analyze or Sync, is analyzed.
    3. Metadata from data sources for which the value of this setting is Sync, is synchronized.

    ValueDescription
    Load

    Harvest metadata from the data source and upload it to your Collibra environment. This allows you to inspect and, if necessary, edit the harvested metadata before uploading it to the Collibra Data Lineage service instance for analysis.

    When the job is done, you can download and review the metadata:

    1. Open the Activities list.
    2. In the row containing the job, click Result.
      The Synchronization Results dialog box appears.
    3. Click download and save the ZIP file to your hard drive.

    Tip The download link resembles the following: https://integrations.collibra-abc.com/rest/2.0/files/01944f12-7665-7d9c-8bc5-aa426b6a63cc. Take note of the file ID, in this example: 01944f12-7665-7d9c-8bc5-aa426b6a63cc. After you inspect the metadata, you can send the ZIP file for analysis by using the "Analyze files" option. Alternatively, you can upload the ZIP file using the POST /files API. In either case, you need to specify the file ID.

    Analyze

    Load and analyze the metadata on the Collibra Data Lineage service instance.

    Synchronization does not start after analysis; it starts only after either:

    Important  If you want to synchronize multiple data sources, we strongly recommend that you select this option in the respective Edge or Collibra Cloud site capabilities for each of your data sources. This allows you to synchronize all data sources in a single job, thereby maximizing efficiency and mitigating the risk of failed synchronization jobs.
    Sync

    Load, analyze, and synchronize metadata from all data sources. Synchronization starts – or is queued, if another synchronization job is running – immediately after analysis.

    Important If you want to synchronize multiple data sources and you select this option, each data source is processed as a separate job. This is highly inefficient and will likely lead to failed sync jobs. For complete information and important considerations, go to Tips for successful lineage synchronization.

    Yes

    Active

    The option determines whether to include or remove the technical lineage of the data source.

    Select this option to include the technical lineage of this data source.

    Clear the checkbox to exclude the technical lineage of this data source.

    No

    Maximum parallel requests

    This optional property allows you to specify the internal sizing, meaning the amount of tasks that can be executed at the same time.

    The default value is "1", which means that HTTP requests are run in a synchronous manner, instead of in parallel. As value of "5", for example, means that as many as 5 HTTP requests can take place in parallel.

    A lower value reduces the chances of experiencing HTTP 401 Unauthorized errors.

    No

    Debug

    This setting is not valid for this integration. It should be set to false.

    No

    Log level

    Only complete this field on the request of or together with Collibra Support.

    No

  5. Click Create.
    The capability is added to the Edge or Collibra Cloud site.
    The fields become read-only.

What's next

You can now synchronize the technical lineage.