Upgrade Collibra Data Quality & Observability

Note Before proceeding with any upgrades, back up your Collibra DQ metastore.

Warning Rolling back Collibra DQ to an older version is not supported. Please contact Collibra Support with any questions.

Download the Collibra DQ Upgrade Package

Note Starting with the December 2021 release, all Collibra DQ customers upgrading or patching receive the Full package, not the Base package, and should follow the same upgrade steps described in this section.

Download the tarball using the signed link to the full package tarball provided by Collibra. Replace <signed-link-to-full-package> with the link provided.

### Go to the OWL_BASE (home directory of the user is most common)
### This example we will use /home/owldq installing as the user owldq

cd /home/owldq 

### Download & untar
curl -o dq-full-package.tar.gz "<signed-link-to-full-package>"
tar -xvf dq-full-package.tar.gz

### Clean-up unnecessary tarball (optional)
rm dq-full-package.tar.gz

Upgrade steps

Important Verify the Collibra Data Quality & Observability version from which you are upgrading before proceeding with the upgrade. To do this, sign into your Collibra DQ instance and click the in the upper-right corner of any page. The App Version lists your current Collibra DQ version. Select an option from the tabs below depending on the version from which you are upgrading and the version to which you plan to upgrade.

Upgrading data source drivers

When new data source drivers are available, they are listed in the Release Notes or recommended to you directly by Collibra. Determine which drivers need to be updated and follow these steps:

1. Review the release notes to see which drivers need to be updated.
2. From the previously extracted tarball provided to you by Collibra, locate the drivers.tar.gz file and extract the contents into a new directory called drivers.
3. Rename your /owl/drivers folder "/owl/old_drivers/" to create a backup of the old drivers.
4. Replace the drivers from the /owl/old_drivers/ folder with the new drivers extracted from the tarball /owl/drivers/.

   Example If you replace an old Databricks driver with a new one, the file path might look like /owl/drivers/databricks.

Deploy the new Collibra DQ Helm Chart on your Kubernetes cluster to upgrade to a new version of Collibra DQ.

1. Download the Docker key.json from the Object Store and run the following command.
Copy

docker login -u _json_key -p "$(cat client-repo-key.json)" https://gcr.io

2. Using the Docker Pull Commands you obtained from the Collibra Downloads page, run one command per container to pull the Collibra DQ containers. The Docker pulls should follow a similar format to the ones in the following example:
Copy

docker pull gcr.io/owl-hadoop-cdh/dq-web:2023.11-ABDGCSHILM-123
docker pull gcr.io/owl-hadoop-cdh/dq-agent:2023.11-ABDGCSHILM-123
docker pull gcr.io/owl-hadoop-cdh/dq-livy:3.4.1-2023.11-ABDGCSHILM-123
docker pull gcr.io/owl-hadoop-cdh/dq-spark:3.4.1-2023.11-ABDGCSHILM-123

3. Tag the pulled containers and upload them to your internal Artifactory for deployment.
4. To deploy the new Collibra DQ Helm Chart on your Kubernetes cluster and upgrade to a new version of Collibra DQ, run the following Helm commands, replacing the placeholder values with the actual values described in the table below.

Important 
The following example command may vary depending on the options your organization supplied during the initial installation.

When upgrading, it's crucial to consider Helm properties provided through extra options or defaulting in Helm Charts to avoid losing any existing configurations.

helm upgrade --install --namespace <namespace> \
--set global.version.dq=<cdq_version> \
--set global.version.spark=<cdq_spark_version> \
--set global.configMap.data.license_key=<cdq_license_key> \
--set global.configMap.data.license_name=<your_license_name> \
--set global.web.admin.email=${email} \
--set global.web.admin.password=${password} \
--set global.web.service.type=ClusterIP \
--set global.image.repo=<pathTolmageRepo> \
<deployment_name> \
/path to the helm chart root folder>

Value	Description
<namespace>	The namespace that you created and into which you deployed Collibra DQ.
<cdq_version>	The version from the web image suffix. For example, 2023.11 from the image, dq-web:2023.11.
<cdq_spark_version>	The Spark version from the Spark image suffix. For example, 3.4.1-2023.11 from the image, spark:3.4.1-2023.11.
<cdq_license_key>	The license key provided to you by Collibra.
<your_license_name>	The license name provided to you by Collibra.
${email}	The default admin user email associated with the admin account.
${password}	The default admin user password for the admin account. The password must adhere to the following password policy: A minimum of 8 characters. A maximum of 72 characters. At least one upper-case character. At least one numeric character. At least one supported special character (!@#%$^&*?_~). Cannot contain the user ID (admin). Note If a password that does not meet the password policy is entered, the install process proceeds as though the password is accepted, but the admin user becomes locked out. If this occurs, rerun the Helm command with a password that meets the password policy and restart the web pod.
<pathTolmageRepo>	This is your private registry key, where the Collibra images are available. When this is not provided, you will pull the images from the Collibra image registry, for which you should create a pull secret with the repo key provided by Collibra. See the Cloud native install documentation for more details about how to create a pull secret.
<deployment_name>	Any name of your choice for this deployment.

5. Check that all required Kubernetes resources are available and running.
   1. List all of the resources of the namespace in your Kubernetes cluster into which Collibra DQ is deployed, including pods, Ingress/LoadBalancer, and statefulsets.

   kubectl get all -n <namespace>

6. Create a test DQ job without any optional DQ layers or rules to verify that the Spark driver, executors, and containers are able to launch successfully.