Airflow: Set up OpenLineage integration and prepare files for Shared Storage connections

Use this procedure to configure your software to emit OpenLineage messages and save the resulting files to a location accessible by Collibra.

1. To install and configure the OpenLineage integration in Airflow, follow this guideline in the Airflow documentation: Using OpenLineage integration.
   

   You can use the following configuration as an example:

   [openlineage]
   
   transport='{"type":"http", "url": "http://HOST_OR_URL_WHERE_FLUENTD_IS:8888/openlineage'
   
   namespace = 'airflow'

2. After Fluentd saves the lineage messages to files, on your Edge site, create a directory and note its path. This path is required when you create a Shared Storage connection.
3. Copy the files in OpenLineage format to the directory that you created in step 2.
   Note Whenever you synchronize lineage, you must upload all source files you want to include in the technical lineage graph.

What's next

You can now create a Shared Storage connection.