Prepare the MicroStrategy <source ID> configuration file (NEW)(Beta)

The lineage harvester uses the configuration file to connect to MicroStrategy. You must also prepare a MicroStrategy <source ID> configuration file to:

  • Specify the default domain, meaning the domain in Collibra in which the corresponding assets of MicroStrategy metadata will be ingested.
  • Optionally, specify from which MicroStrategy projects you want to ingest metadata.
  • Optionally, configure data source mapping, to map the name of a data source returned by the lineage harvester to the true name of the data source.

Tip "<source ID>" refers to the value of the Id property in the lineage harvester configuration file.

Example 
.

Copy code 

{
    "default_domain_id": "1a0a942e-e3a7-45a1-83e8-ade30b1cab1a",
    "filters": [
        {
            "projectIds": [],
            "projectNames": ["Customers","Research","Marketing"]
        }
    ],
    "datasourceMapping": [
	 {
	     "found_datasource": "REDSHIFT",
	     "found_project": "*",
	     "mapping": {
		  "dbname": "RD_pearl",
		  "schema": "Default_North",
		  "dialect": "spark",
		  "collibraSystemName": "TV_dev"
	     }
	 }
    ]
}

Tip Click Copy code to copy the example to your clipboard.

Steps

  1. Create a new JSON file in the lineage harvester config folder.
  2. Give the JSON file the same name as the value of the Id property in the lineage harvester configuration file.
    Example If the value of the Id property in the lineage harvester configuration file is mstr-source-1, then the name of your JSON file should be mstr-source-1.conf.
    Important Your JSON file must have the file extension .conf.
  3. For each database in MicroStrategy, add the following content to the JSON file:
    Tip You can use wildcards to capture multiple string combinations for any of these properties.
    PatternDescription
    *Matches everything.
    ?Matches any single character.
    [seq]Matches any character in "seq".
    [!seq]Matches any character not in "seq".

    Property

    Description

    Mandatory

    default_domain_id

    The domain in which you want the corresponding assets of MicroStrategy metadata to be ingested.

    Yes

    filters

    This section allows you to specify from which MicroStrategy projects you want to harvest metadata.

    All metadata is ingested into the default domain, as specified via the default_domain_id property.

    If you don't want to filter on projects, don't include this section in your <source ID> configuration file.

    No

    projectIds
    		The IDs of the MicroStrategy projects from which you want to ingest metadata.

    No

    projectNames
    		The project names of the MicroStrategy projects from which you want to ingest metadata.

    No

    datasourceMapping

    This optional section allows you to configure data source mapping. Include this section only if you need to differentiate between multiple data sources that have the same name.

    No

    found_datasource

    The name of the data source that was returned by the lineage harvester, as shown in the technical lineage.

    Note The data source name is case-sensitive.

    Yes

    found_project

    The name of the project in which the data source information resides. You can specify an asterisk (*) to search for data source information across all projects.

    Yes

    mapping

    Use this section to map the data source name that was returned by the lineage harvester to the true name of the data source.

    Example You have a Redshift data source named "RD_pearl", but the lineage harvester has returned the name "Redshift_connection". You can configure the datasourceMapping section as follows: 
    {
    "datasourceMapping": [
	 {
	     "found_datasource": "REDSHIFT",
	     "found_project": "*",
	     "mapping": {
		  "dbname": "RD_pearl",
		  "collibraSystemName": "TV_dev"
	     }
	 }
    ]
}

    Yes

    dbname

    The name of the database to which you want to map the found data source.

    Yes

    schema

    The name of the schema in MicroStrategy.

    No

    dialect

    The dialect of the data source in MicroStrategy.

    No

    collibraSystemName

    The system or server name of a database.

    If you set the useCollibraSystemName property to true in your lineage harvester configuration file, but you either don't create a <source ID> configuration file, or don't specify a value for the collibraSystemName property in your <source ID> configuration file, the system name in the technical lineage is "DEFAULT".

    If you set the useCollibraSystemName property to false in your lineage harvester configuration file, leave this property empty as follows: "collibraSystemName": "".

    Let's assume that you have a data source named Customers. You use this data source connection in two different projects, Project_A and Project_B, but they are actually two different databases. When you prepare the physical data layer in Data Catalog, you create a System asset for each of these databases. Let's say you named them Customers-North and Customers-South. You can then configure this property as follows.

    "datasourceMapping": [
        {
            "found_datasource": "Customers",
            "found_project": "Project_A",
            "mapping": {
                "dbname": "Customers",
                "collibraSystemName": "Customers_North"
            }
        },
        {
            "found_datasource": "Customers",
            "found_project": "Project_B",
            "mapping": {
                "dbname": "Customers",
                "collibraSystemName": "Customers_South"
            }
        }
    ]

    Warning The values of this property must exactly match the name of your System asset in Collibra.

    Yes

  4. Save the <source ID> configuration file.

What's next?

Run the lineage harvester. For the required commands, see Overview MicroStrategy integration steps (NEW, beta).