Airflow: Set up Fluentd for cloud storage

Use the following steps to configure Fluentd to receive and store OpenLineage events.

Steps

1. Determine the host and port where the Fluentd collector will run. This daemon needs to be up and running continuously to collect metadata.
2. Open a port for HTTP so that the data source can send REST API events to this location. Ensure that the port is open for listening, and document the port number and host.
   

   You can use HTTPS for improved security. Generate an SSL certificate and configure Fluentd when using HTTPS. For more information, go to http in Flentd documentation.

3. Download the Fluent package by following the guide for your system. For example, if you use Red Hat Linux, go to Install by RPM Package (Red Hat Linux) in the Fluentd documentation.
4. Start the service and check the status. For example, you can enter the following command:
   - sudo systemctl start fluentd.service; sudo systemctl status fluentd.service
5. Check the logs to ensure the service is running correctly. For example, if you use Red Hat Linux, you can find the log files at /var/log/fluent/fluentd.log.
6. Test basic functionality by completing the following steps. Red Hat Linux is used as an example.
   1. The default configuration is set to receive logs at an HTTP endpoint on port 8888 and route them to stdout. On Red Hat Linux, the default configuration is at /etc/fluent/fluentd.conf.
   2. Enter the following command to test:
      curl -X POST -d 'json={"json":"Here is my message"}' http://localhost:8888/debug.test
   3. Check the outcome by viewing the latest log entry:
      tail -n 1 /var/log/fluent/fluentd.log
7. Configure Fluentd for OpenLineage.

   The following sample configuration shows how to collect messages on port 8888 and save them to files grouped by 5-minute windows. For details, use Configuration in the Flentd documentation.

   <source>
     @type http
     port 8888
     json_array true
   </source>
   <match openlineage>
     @type file
     path /fluentd/logs/openlineage
     path_suffix .jsonl
     json_array true						
     <buffer>
       timekey 5m
       timekey_use_utc true
       timekey_wait 5m
     </buffer>
     append true
     <format>
       @type json
     </format>
   </match>

   You can test the result after configuration by entering the following command:
   curl -X POST -d 'json={"json":"Here is my message"}' http://localhost:8888/openlineage

   And then check the outcome by entering the following command:
   ls -l /var/log/fluent/collect_packets

What's next

Airflow: Set up OpenLineage integration and prepare files for Cloud Storage connections.