Configure Spark scratch disk space to handle large datasets

When you run large datasets, your Spark cluster's resources need to be sized appropriately to handle the computation of DQ rules. This section shows you how to configure on-demand Spark scratch disk space to handle huge Spark jobs.

Prerequisites

Before you configure Spark scratch disk space, ensure you have:

  • A Collibra DQ deployment on a cloud native platform, such as EKS, GKS, and so on.
  • A Kubernetes service account for Collibra DQ that meets the permission requirements to provision dynamic Persistent Volume Claims (PVCs) in ReadWriteOnce (RWO).
  • Execute permissions on the /tmp directory or the directory used for temporary storage, to ensure Spark jobs launch successfully.

Note This capability is only available for Collibra DQ versions 2023.02 or newer.

Steps

  1. Open a terminal session.
  2. Set the following values to deploy or update the Collibra DQ app with the provided Helm charts:
    Copy
    --set global.configMap.data.spark_scratch_type="persistentVolumeClaim" --set global.configMap.data.spark_scratch_storage_class="standard" --set global.configMap.data.spark_scratch_storage_size="20Gi" --set global.configMap.data.spark_scratch_local_path="/tmp/scratch"

    3. Note The values in the previous example are settings for GKE-based deployments.

    Important If you use Snowflake key pair authentication for Pullup jobs, the Spark scratch local path and the private key staging directory must not be identical. The staging directory is OWL_SCRATCH_DIR if set, otherwise the JVM temporary directory, which defaults to /tmp/scratch. If both resolve to /tmp/scratch, Kubernetes rejects the Spark driver pod with the following error: spec.containers[0].volumeMounts[1].mountPath: Invalid value: "/tmp/scratch": must be unique

    To avoid this, set spark_scratch_local_path to a different value, for example /tmp/spark-scratch, by running a Helm upgrade against your existing Collibra DQ deployment: helm upgrade ... --set global.configMap.data.spark_scratch_local_path="/tmp/spark-scratch"

    After the upgrade completes, rerun the affected Pullup job and confirm that the Spark driver pod is created. For more information, go to Connecting to Snowflake.

    1. If your Collibra DQ app is managed by an alternative Kubernetes package manager, ensure you update the owl-agent-configmap with the following variables:
      2. Copy 
      KUBERNETES_SCRATCH_TYPE: persistentVolumeClaim
      KUBERNETES_SCRATCH_STORAGE_CLASS: standard
      KUBERNETES_SCRATCH_STORAGE_SIZE: 20Gi
      KUBERNETES_SCRATCH_LOCAL_PATH: /tmp/scratch

      Note Pick the value of Storage Class, Storage Size and mouthPath specific to your requirements.

    2. Restart the DQ agent pod to complete the updates.
  3. Sign in to Collibra DQ.
  4. Hover your cursor over the icon and click Admin Console.
    The Admin Console opens.
  5. Click Agent Configuration.
    The Agent Configuration page opens.
  6. Click the Actions, then click Edit icon Edit.
    The Edit Agent modal opens.
  7. In the Free form (Appended) text box option, enter the following string value:
    8. Copy 
    -conf spark.kubernetes.driver.podTemplateFile=local:///opt/owl/config/k8s-driver-template.yml,spark.kubernetes.executor.podTemplateFile=local:///opt/owl/config/k8s-executor-template.yml,spark.kubernetes.driver.reusePersistentVolumeClaim=false

    Tip If you add any additional configuration properties based on your specific requirements, separate them with commas.

  8. Optionally, increase driver pod memory. This may be needed for large datasets, if the driver pod is crashing or jobs end up in Unknown State. Use the following Spark command:
    9. Copy 
    -conf spark.kubernetes.memoryOverheadFactor=0.4
  9. Click Submit.
  10. Run a sample DQ job to process DQ rules against a dataset of your choice. The corresponding Spark job dynamically provisions PVCs for the driver and each executor to handle large datasets by juggling between Memory and Disk at the mouth path of the PVC.

What's next

For more information, go to the official Spark documentation on: