Configure job limits

Limits allow admins to control data thresholds globally across Collibra to ensure job runs are consistent, relevant, and efficient.

Prerequisites

You have the Data Quality Admin global role with the Data Quality > Manage Settings global permission.

Steps

  1. On the main toolbar, click Products icon → Settings.
  2. Click the Data Quality tab.
  3. Click Limits.
  4. Update the limits, as needed.
    5. Limit Description
    Lookback limit

    The lookback limit is the number of previous job runs that Data Quality & Observability examines to train the adaptive rules model.

    The default value is 21. You can adjust it by entering a value greater than 0.
    Score thresholds

    Score thresholds are specific data quality scores that you can configure to determine whether a job is passing or failing. Scores between the passing and failing thresholds are marked as warnings to alert users to possible or emerging data quality issues.

    Set the custom scoring thresholds using the Max breaking and Min passing fields.
    Job log retention period

    The number of days the Data Quality Job execution logs are retained before being removed.

    The default is 3 days.
    Platform log retention period

    The number of days the Data Quality & Observability (cloud) platform-level logs are retained before being removed.

    The default is 1 day.
    Maximum concurrent pushdown jobs

    The number of Pushdown jobs that can run at the same time.

    The default is 5. You can adjust it by entering a value between 1 and 10. Jobs started through the API always stay capped at 10, even if this setting is increased.

    Contact Collibra Support if you need to increase this limit beyond 10.
    Maximum concurrent pullup jobs

    The number of Pullup jobs that can run at the same time.

    The default is 5. You can adjust it by entering a value between 1 and 25.
    Pullup job sizing limits

    The maximum Spark resources each Pullup job can request.

    For recommendations on how you might consider configuring these settings, go to Pullup job sizing recommendations.
    Maximum number of executors

    The maximum number of executors that the Spark agent can allocate to a job to process it efficiently.

    The default is 4.
    Maximum executor memory

    The maximum amount of RAM per executor that the Spark agent can allocate to a job to process it efficiently.

    The default is 8 GB.
    Maximum number of driver cores

    The maximum number of driver cores that the Spark agent can allocate to a job to process it efficiently.

    The default is 1.
    Maximum driver memory

    The maximum amount of driver memory that the Spark agent can allocate to a job to process it efficiently.

    The default is 2 GB.
    Maximum executor cores

    The maximum number of executor cores that the Spark agent can allocate to a job to process it efficiently.

    The default is 4.
    Total number of workers

    The total number of Spark workers available for a pullup job to use. In Spark Standalone, this is the maximum number of Spark workers in your cluster. In Kubernetes, this is the maximum number of VMs. This value is proportionate to the Maximum worker memory and Maximum worker cores settings.

    The default is 10.
    Maximum worker memory

    The maximum amount of memory per Spark worker that the Spark agent can allocate to a job to process it efficiently. This value is proportionate to the total number of workers in your environment.

    The default is 8 GB.
    Maximum worker cores

    The maximum number of CPU cores per worker that the Spark agent can allocate to a job to process it efficiently. This value is proportionate to the total number of workers in your environment.

    The default is 4.
    Maximum partitions

    The maximum number of partitions in your data to allow Spark to process large jobs more efficiently.

    A partition is an even split of the total number of rows in your data. For large jobs, increasing the number of partitions can improve performance and processing efficiency. If your data is unevenly distributed, you may need to increase the number of partitions to avoid data skew.

    The default is 100.
    Parallelism factor
    		 The parallelism factor setting that controls the Spark job execution parallelism. This setting accepts values from 1 to 8, allowing you to optimize performance by increasing partition density per executor unit.
  5. Click Save changes.

Helpful resources

What's next