Airflow: Set up OpenLineage integration for Shared Storage connections

Use this procedure to configure your software to emit OpenLineage messages and save the resulting files to a location accessible by Collibra.

1. To install and configure the OpenLineage integration in Airflow, follow this guideline in the Airflow documentation: Using OpenLineage integration.
   

   You can use the following configuration as an example:

   [openlineage]
   
   transport='{"type":"http", "url": "http://HOST_OR_URL_WHERE_FLUENTD_IS:8888/openlineage'
   
   namespace = 'airflow'

2. After Fluentd saves the lineage messages to files, on your Edge site, create a directory and note its path.
3. Copy the files in OpenLineage format to the directory that you created in the previous step.
   Note Whenever you synchronize lineage, you must upload all source files you want to include in the technical lineage graph.

What's next

You can now set up Fluentd and prepare the data source files