Overriding the default batch size

Unless you set the batch size in the owl-env.sh script for Standalone deployments or the Helm chart for Cloud native deployments, the batch size defaults to 5. This means that the DQ Agent applies a limit of 5 Jobs that can run concurrently. This page shows you how to override the default batch size for both Standalone and Cloud native deployments.

  • Standalone
  • Cloud native
  1. Open a terminal session.
  2. Open the owl-env.sh file with the following command:
    3. Copy 
    vi /home/owldq/owl/config/owl-env.sh
    export DEFAULT_BATCH_ID=<your-b64enc-batch-id>
  3. Use a base64 encoder, such as the free encoder from base64encode.org, to encode your batch ID and enter the encoded value after DEFAULT_BATCH_ID.

    4. Tip Your base64 encoded string should look similar to the following example:

    export DEFAULT_BATCH_ID=QUdFbVlYWnNkRDRpWVpiZXZGTnBlbVU5Tkw9PQ==

  4. Enter the following command:
    5. Copy 
    export DEFAULT_BATCH_SIZE=<your-batch-size>

    Tip Enter a value in <your-batch-size> that meets the concurrency sizing requirements of your use case.

  5. End the terminal session.
  1. Open a terminal session.
  2. Enter the following command to pass your new default values to your agent batch ID and agent batch size:
    3. Copy 
    --set global.configMap.data.agent_batch_id="default?batchSize=<your-b64enc-batch-id>" 
    --set global.configMap.data.agent.data="<your-batch-size>"

    Tip Enter a value in <your-batch-size> that meets the concurrency sizing requirements for your use case.

    1. If your Collibra DQ app is managed by an alternative Kubernetes package manager, ensure you update the owl-agent-configmap with the following variables:
      2. Copy 
      AGENT_BATCH_ID: "your-base64-encoded-value"
      AGENT_BATCH_SIZE: <your-new-batch-size>

      Tip Your base64 encoded string should look similar to the following example:

      export DEFAULT_BATCH_ID=QUdFbVlYWnNkRDRpWVpiZXZGTnBlbVU5Tkw9PQ==

      Tip Enter a value in <your-batch-size> that meets the concurrency sizing requirements for your use case.

    2. Restart the DQ agent pod to complete the updates.
  3. Choose one of the following options to update your DQ agent resources:
    1. Update the DQ agent from the user interface
      1. Sign in to Collibra DQ.
      2. Hover your cursor over the icon and click Admin Console.
        The Admin Console opens.
      3. Click Remote Agent.
        The Agent Management page opens.
      4. Click the pencil icon.
        The Edit Agent modal opens.
      5. Update Number of Executors, Executor Memory, Number of Cores, and Driver Memory according to the concurrency sizing requirements for your use case.
      6. Click Save.
    2. Update the DQ agent through Helm to set your new limits
      1. Open a terminal session.
      2. In your Helm values.yaml file, update the DQ agent's memory and cpu quotas according to the concurrency sizing requirements for your use case:
        3.  agent:
            enabled: true
            replicaCount: 1
            nodeSelector:
              key: ""
              value: ""
            resources:
              requests:
                memory: "<your-new-memory-value>"
                cpu: "<your-new-cpu-value>"
              limits:
                memory: "<your-new-memory-value>"
                cpu: "<your-new-cpu-value>"
      3. Perform the Helm upgrade on the existing DQ release.
      4. End the terminal session.