Securing Passwords

Collibra DQ stores passwords in encrypted format in the PostgreSQL metastore using an industry standard encryption algorithm, which combines SHA-512 hashing with AES-256 encryption for storing passwords (encryption algorithms are listed in Overview). Usernames are stored in plain text for reference.

Encryption is applied automatically when credentials are saved, either through the user interface or CLI. Currently, the encryption key is static and embedded in the application source code. Customers cannot override it directly.

Passwords are decrypted in memory by the DQ Agent only when needed to establish a database connection. For example, during dataset creation or job execution.

If using a password manager, the retrieved credentials are cached in memory for approximately 60 minutes or until job execution ends. They are not persisted anywhere in the DQ metastore.

Encrypting passwords from the command line

To avoid sending plain text passwords when you run DQ jobs from the command line, you can encrypt passwords instead. To encrypt your password, execute the following command:

owlmanage.sh encrypt=password

The output password should look similar to the following example:

Q+Ri1S+ljpG+fDefXLY4/vXtUosspAoL

You can use this password in any DQ job from the command line where you would normally use a plain text password.

The following is an example of a DQ job with an encrypted password instead of a plain text password:

./owlcheck -q "SELECT id, browser->'$.name' browser FROM events" -c "jdbc:mysql://54.212.36.218:2212/test" -u "owl" -p "Q+Ri1S+ljpG+fDefXLY4/vXtUosspAoL" -driver "com.mysql.cj.jdbc.Driver" -lib "/opt/owl/drivers/mysql8" -ds jsonremotemysql -rd "2022-07-25"

If you run a DQ job from within the DQ Web UI, it automatically encrypts your password, eliminating the need to manually encrypt it. For added security, all passwords are masked in the logs and plain text passwords are never stored.

Securing passwords in a Kubernetes environment

To ensure that your password is secure in a Kubernetes environment, you must use Kubernetes Secrets. A Secret stores sensitive data, for example, a username or password, which are used by pods to access services.

You can create the Secret by using either of the following methods:

• Pass the raw data in the command.
• Store the credentials in files that you pass in the command.

The following commands create a Kubernetes Secret that stores the username admin and the password S!B\*d$zDsb=.

Create Secrets using raw data

Execute the following command:

kubectl create secret generic db-user-pass \
                --from-literal=username=admin \
            --from-literal=password='S!B\*d$zDsb='

Note To prevent your shell from interpreting special characters, such as $, \, *, =, and ! in your strings, you must use single quotes '' to escape those characters.

Create Secrets using source files

You can create Kubernetes Secrets by using source files.

Steps

1. Store the sensitive data/credentials in files:

echo -n 'admin' > ./username.txt
            echo -n 'S!B\*d$zDsb=' > ./password.txt

In this command, the -n flag ensures that the generated files have no newline characters at the end of the text. When kubectl reads a file and then encodes the content into a base64 string, the extra newline character also gets encoded. You don't need to escape special characters in strings that you have included in a file.

2. Pass the file paths in the kubectl command:

kubectl create secret generic db-user-pass \
                --from-file=./username.txt \
            --from-file=./password.txt

The file name in this case is the default key name. You can also set the key name using the following command:

kubectl create secret generic db-user-pass \
                --from-file=username=./username.txt \
            --from-file=password=./password.txt

With either method, the output is similar to the following:

secret/db-user-pass created

Verify the Kubernetes Secret

Now that you have created the Secret, you should verify that it was created with the following command:

kubectl get secrets

The output is similar to the following:

NAME             TYPE            DATA            AGE
            db-user-pass     Opaque          2               51s

Now, view the details:

kubectl describe secret db-user-pass

The output is similar to the following:

Name:            db-user-pass
                Namespace:       default
                Labels:          <none>
                Annotations:     <none>

                Type:            Opaque

                Data
                ====
                password:    12 bytes
            username:    5 bytes

By default, the kubectl get and kubectl describe commands avoid showing the contents of a Secret. This protects the Secret from being exposed by mistake or from being stored in a terminal log.

Decode the Kubernetes Secret

You should now decode the Secret.

Steps

1. View the contents of the secret with the following command:

kubectl get secret db-user-pass -o jsonpath='{.data}'

The output is similar to the following:

{ "password": "UyFCXCpkJHpEc2I9", "username": "YWRtaW4=" }

2. Decode the password data with the following command:

echo 'UyFCXCpkJHpEc2I9' | base64 --decode

The output is similar to the following:

S!B\*d$zDsb=

Warning This particular example is strictly for documentation purposes only. This method could cause the command with the encoded data to be stored in your shell history, meaning that anyone with access to your computer could potentially locate the command and decode the secret. Therefore, you should combine the view and decode commands, as shown in the following command:

kubectl get secret db-user-pass -o jsonpath='{.data.password}' | base64 --decode