Logging best practices

Recommendation

By default, Collibra maintains a 30-day archive of log files. The following recommendations will help in out-of-the-ordinary circumstances where you need to go beyond the default settings.

Impact

• Comply with industry-specific log retention requirements
• Keep your environment performant, avoiding potential causes of:
  • Issues with restores
  • Issues with Import jobs

Recommendations

• When setting up JDBC logs, set parameters to follow the same folder structure as Jobserver logs to ensure they are written to locations with the appropriate permissions and are easy to locate.
• In general operation, the default logging level is usually sufficient. Only when working with Collibra Support or Engineering teams to troubleshoot specific issues should you set logging levels to higher levels of detail (for example DEBUG or TRACE). This should only be maintained until the issue has been duplicated and captured in the logs and then the logging level should be returned to the default because if higher levels of logging are maintained too long, they can inadvertently exhaust the available disk space faster than monitoring tools can detect.
• By default, the system retains 30 days of logs in the archive. So, if there is a business or regulatory requirement to retain log files for longer than 30 days, we recommend external scripting to call the Collibra Support REST API (<base console url>/docs/rest/index.html#/support) to move archived files to a non-Collibra storage space, such as an S3 bucket, every 2 weeks.

Validation criteria

Use Collibra Console → Settings → Logs to monitor these recommendations.

For more information see Collibra's Documentation Center (requires Collibra login):

• Logging
• Environment log settings for DGC services
• Environment log settings for Repository services