Adding JDBC and remote file connections

This topic shows you how to create a new data source connection.

Prerequisites

To add a connection, you need:

  • A Collibra DQ account with ROLE_ADMIN or ROLE_CONNECTION_MANAGER permissions.
  • Access to the account credentials of your connection, for example, PostgreSQL.

Adding a JDBC or remote file connection

This section shows you how to connect to a JDBC or remote file data source from the Connection Management page. Select an option from the tabs below.

  • JDBC Connections
  • Remote File Connections
  1. Sign into Collibra DQ.
  2. Hover your cursor over the icon and click Admin Console.
    The Admin Console opens.
  3. Click Connections.
    The Connection Management page opens.
  4. Click Add Connection on the upper right side of the page.
    The Add New Connection dialog opens to the JDBC Connections tab with a list of supported JDBC data sources.
  5. Click a data source.
    The Add Connection template appears.
  6. Enter the required information.
    • Details
    • Properties
    • Variables
    Required Option Type Value
    Connection Details

    Yes

    		Name String The unique name used for your connection.

    Yes

    		Connection URL String

    The connection string value of your connection. The Connection URL uses the following format:

    jdbc:<datasource>://$host:<port>/<AdditionalProperties>

    Replace the Connection URL to point to the database server to run DQ Jobs against.

    Yes

    		Port Integer The port for the connection to the data source host.

    No

    		Limit Schemas Option

    Allows you to manage usage and restrict visibility to only the necessary schemas in the Explorer tree.

    See Limiting schemas to learn how to limit schemas from the Connection Management page.

    Note When you include a restricted schema in the query of a DQ Job, the query scope may be overwritten when the job runs. While only the schemas you selected when you set up the connection are shown in the Explorer menu, users are not restricted from running SQL queries on any schema from the data source.
    Connection Credentials

    Yes

    		Authentication Type Option

    The method to authenticate your connection.

    Note The configuration requirements differ depending on the authentication type you select. Additional options on the connection template, such as Username and Password, will appear depending on the authentication type you select.

    No

    		Pushdown Option

    Enables Pushdown processing. This option is required for Pushdown Job submission.

    Note This option is currently only available for the connections listed on Pushdown processing.

    No

    		Source Name String

    The alias name of your connection. While this is optional, you can use this to make it easier to identify your connection in API payloads.

    Tip There are no character restrictions for the Source Name field, but it is best practice to use only alphanumeric characters, hyphens, and underscores.

    No

    		Target Agents String The Agent used to submit your DQ Job.
    Required Property Type Value

    Yes

    		Driver Location File path

    The location where your driver file is saved.

    Click Upload to specify the Directory Name and select a Driver File.

    Yes

    		Driver Name String

    The driver class name.

    For example, a BigQuery JDBC driver may use the following convention:

    com.simba.googlebigquery.jdbc42.Driver

    No

    		Driver Properties String

    The configurable driver properties of your connection.

    The supported format is [{"name":"prop1","value":"val1"},{"name":"prop2","value":"val2"}]
    Required Property Type Value

    Yes

    (when using)

    		 Variable Name String

    The variable to display in place of the sensitive information. This can be any variable, such as Password

    In the Connection URL, replace the sensitive information with this value. In the following example, dquser and dqpassword are the property names that replace the sensitive variable values: jdbc:postgresql://localhost:5432/postgres?user=${dquser}&password=${dqpassword}

    Yes

    (when using)

    		 Value String

    The sensitive value to hide in the Connection URL. For example, if the password value of the connection credential is "password123", enter password123 in the property value field.

    Tip Commas and equals signs are supported characters. For example, val1=,val2 is a valid format.

  7. Click Submit.
  1. Sign into Collibra DQ.
  2. Hover your cursor over the icon and click Admin Console.
    The Admin Console opens.
  3. Click Connections.
    The Connection Management page opens.
  4. Click Add Connection on the upper right side of the page.
    The Add New Connection dialog opens to the JDBC Connections tab
  5. Click the Remote File Connections tab.
    6. The Add New Connection dialog opens to the JDBC Connections tab with a list of supported remote file data sources.
  6. Click a data source.
    The Add Connection template appears.
  7. Enter the required information.
    • Details
    • Properties
    • Variables
    Required Connection Detail Type Value

    Yes

    		Name String The unique name used for your connection.

    Yes

    		Connection URL String

    The connection string value of your connection.

    Example s3://dq-datasets-123

    Yes

    		Authentication Type Option

    The method to authenticate your connection.

    Note The configuration requirements differ depending on the authentication type you select. Additional options on the connection template, such as Keytab, will appear depending on the authentication type you select.

    No

    		Target Agents String The Agent used to submit your DQ Job.

    No

    		Archive Breaking Records Option Select this option to automatically export CSV files containing the breaking records of DQ Job to your remote file data source.

    No

    		Archive Location String

    The path along which break records will be archived when the Archive Break Records and HTTPS options are enabled.

    Specify a schema output location in your data source for break records to send. For example, /write/FolderName

    Note This option is only available when the HTTPS option is enabled.
    Required Property Type Value

    No

    		Driver Properties String

    The configurable driver properties of your connection.

    The supported format is [{"name":"prop1","value":"val1"},{"name":"prop2","value":"val2"}]
    Required Property Type Value

    Yes

    (when using)

    		 Variable Name String

    The variable to display in place of the sensitive information. This can be any variable, such as Password

    In the Connection URL, replace the sensitive information with this value. In the following example, dquser and dqpassword are the property names that replace the sensitive variable values: jdbc:postgresql://localhost:5432/postgres?user=${dquser}&password=${dqpassword}

    Yes

    (when using)

    		 Value String

    The sensitive value to hide in the Connection URL. For example, if the password value of the connection credential is "password123", enter password123 in the property value field.

    Tip Commas and equals signs are supported characters. For example, val1=,val2 is a valid format.

  8. Click Submit.

What's next?

Map data source connections to a DQ agent

Known Limitations

  • When you update the name of a connection, a separate connection with the updated name is created in addition to the old connection. Because Collibra DQ treats this as a new connection, the Jobs and details from the old connection do not persist to the new one.
  • When you edit an existing connection and its details, another connection is created and any sensitive properties from the old connection do not persist to the new connection.
    • A possible workaround is to recreate the sensitive properties from the old connection in the new connection.