Prepare the lineage harvester configuration file for MicroStrategy

You have to prepare a configuration file before you run the lineage harvester. The lineage harvester collects your MicroStrategy metadata and sends it to the Collibra Data Lineage service, where it is processed and analyzed. Collibra Data Intelligence Cloud then imports the MicroStrategy assets and relations to Data Catalog.

Before you begin

Set up the latest lineage harvester.
Create a BI Catalog domain in which you want to ingest the MicroStrategy

Requirements and permissions

Collibra Data Intelligence Cloud.
A global role with the following global permissions:
- Catalog, for example Catalog Author
- Data Stewardship Manager
- Manage all resources
- System administration
- Technical lineage
A resource role with the following resource permission on the community level in which you created the BI Data Catalog domain:
- Asset: add
- Attribute: add
- Domain: add
- Attachment: add

Steps

Start the lineage harvester to create an empty lineage harvester configuration file by entering the following command:
- Windows: .\bin\lineage-harvester.bat
- For other operating systems: chmod +x bin/lineage-harvester and then bin/lineage-harvester
An empty configuration file is created in the config folder.

Open the lineage-harvester.conf file and enter the values for each property.

Properties	Description
general	This section describes the connection information between the lineage harvester and Data Catalog.
catalog	This section contains information that is necessary to connect to Data Catalog.
url	The URL of your Collibra Data Intelligence Cloud environment. Note You can only enter the public URL of your Collibra DGC environment. Other URLs will not be accepted.
username	The username that you use to sign in to Collibra.
useCollibraSystemName	By default, the `useCollibraSystemName` property is set to `false`. This property is not valid for MicroStrategy integration. We recommend that you leave this property set to `false`.
sources	This section contains all MicroStrategy connection properties.
type	The kind of data source. In this case, the value has to be `MicroStrategy`.
id	The unique ID of your MicroStrategy metadata. For example, `my_microstrategy`. Warning In the `sources` section of your lineage harvester configuration file, you can only specify one `id` property per MicroStrategy Intelligence Server. If you have multiple `id` properties for a single MicroStrategy Intelligence Server, ingestion will fail. If you have multiple `id` properties in the configuration file, it means you intend to ingest from multiple unique MicroStrategy Intelligence Servers. Tip This value can be anything as long as it is unique and human readable. The ID identifies the batch of MicroStrategy metadata on the Collibra Data Lineage service.
domainId	The unique reference ID of the domain in Collibra Data Intelligence Cloud in which you want to ingest the MicroStrategy assets.
username	The username that you use to sign in to MicroStrategy.
hostname	The endpoint that you use to access the PostgreSQL repository or remote data source, depending on where you installed the lineage harvester. For example `remote.postgres.com`.
port	The port number.
databaseName	Optionally, the name of your database. For example `poc_metadata`.
deleteRawMetadataAfterProcessing	The lineage harvester harvests metadata from specified data sources and uploads it in a ZIP file to a Collibra Data Lineage service instance, for processing. You can use this optional property to specify whether or not the raw metadata should be deleted after it has been processed. The default value is `false`. If the property is set to `true`, the raw metadata is deleted after processing. If set to `false`, it is stored in an Amazon S3 bucket. Note Setting this property to `true` can negatively impact performance. This property is not yet supported by the technical lineage backend, so it can't be used yet. Backend support is coming soon.

Save the configuration file.
Start the lineage harvester again in the console and run the following command:
- for Windows: .\bin\lineage-harvester.bat full-sync
- for other operating systems: ./bin/lineage-harvester full-sync
When prompted, enter the password or client secret to connect to your Collibra Data Intelligence Cloud and MicroStrategy environment.
The passwords are encrypted and stored in /config/pwd.conf

Example

The following example shows a configuration file for MicroStrategy.

{
 "general": {
   "catalog": {
     "url": "https://<organization>.collibra.com",
     "userName": "<your-collibra-username>"
	},
   "useCollibraSystemName": false
 },
  "sources": {
    "type": "Microstrategy",
    "id": "microstrategy-batch",
    "domainId": "<domain-resource-id>",
    "username": "mstr",
    "hostname": "remote.postgres.com",
    "port": 5432,
    "databaseName": "poc_metadata",
	 "deleteRawMetadataAfterProcessing": true
  }
}