Overriding the default batch size

Unless you set the batch size in the owl-env.sh script for Standalone deployments or the Helm chart for Cloud native deployments, the batch size defaults to 5. This means that the DQ Agent applies a limit of 5 Jobs that can run concurrently. This page shows you how to override the default batch size for both Standalone and Cloud native deployments.

  • Standalone
  • Cloud native
  1. Open a terminal session.
  2. Open the owl-env.sh file with the following command:
    3. Copy 
    vi /home/owldq/owl/config/owl-env.sh
  3. Use a base64 encoder, such as the free encoder from base64encode.org, to encode the string default?batchSize=5 (replacing 5 with your desired value) and enter the encoded value after DEFAULT_BATCH_ID in the following command:
    4. Copy 
    # The string "default?batchSize=5" needs to be base64 encoded
    export DEFAULT_BATCH_ID=default?batchSize=5

    Example Your base64 encoded value should resemble the following example:

    export DEFAULT_BATCH_ID=ZGVmYXVsdD9iYXRjaFNpemU9NQ==

  4. Enter the following command:
    5. Copy 
    export DEFAULT_BATCH_SIZE=<your-batch-size>  

    Tip The value in <your-batch-size> should meet the concurrency sizing requirements of your use case.

  5. End the terminal session.
  1. Open a terminal session.
  2. Enter the following command to pass your new default values to your agent batch ID and agent batch size:
    3. Copy 
    --set global.configMap.data.agent_batch_id="default?batchSize=<your-batch-size>" 
    --set global.configMap.data.agent.data="<your-batch-size>"

    Tip Enter a value in <your-batch-size> that meets the concurrency sizing requirements for your use case.

    1. If your Collibra DQ app is managed by an alternative Kubernetes package manager, ensure you update the owl-agent-configmap with the following variables:
      2. Copy 
      AGENT_BATCH_ID: "your-base64-encoded-value"
      AGENT_BATCH_SIZE: <your-batch-size>

      Example Your base64 encoded string should resemble the following example:

      export AGENT_BATCH_ID=QUdFbVlYWnNkRDRpWVpiZXZGTnBlbVU5Tkw9PQ==

      Tip Enter a value in <your-batch-size> that meets the concurrency sizing requirements for your use case.

    2. Restart the DQ agent pod to complete the updates.
  3. Choose one of the following options to update your DQ agent resources:
    1. Update the DQ agent from the user interface
      1. Sign in to Collibra DQ.
      2. Hover your cursor over the icon and click Admin Console.
        The Admin Console opens.
      3. Click Remote Agent.
        The Agent Management page opens.
      4. Click the pencil icon.
        The Edit Agent modal opens.
      5. Update Number of Executors, Executor Memory, Number of Cores, and Driver Memory according to the concurrency sizing requirements for your use case.
      6. Click Save.
    2. Update the DQ agent through Helm to set your new limits
      1. Open a terminal session.
      2. In your Helm values.yaml file, update the DQ agent's memory and cpu quotas according to the concurrency sizing requirements for your use case:
        3.  agent:
            enabled: true
            replicaCount: 1
            nodeSelector:
              key: ""
              value: ""
            resources:
              requests:
                memory: "<your-new-memory-value>"
                cpu: "<your-new-cpu-value>"
              limits:
                memory: "<your-new-memory-value>"
                cpu: "<your-new-cpu-value>"
      3. Perform the Helm upgrade on the existing DQ release.
      4. End the terminal session.