Databricks configuration fields

Use the following table to understand some of the fields that are available for configuring the Databricks data source in Data Access.

Note 
  • The Exclude and Include filters in the following table match against an object's name and support regular expressions. Note that this differs from the Metastore field under Metastore Workspace Pairs, which requires the metastore's unique ID rather than its display name.
  • The Exclude and Include filters apply independently at each object level. To scope ingestion to a specific catalog within a specific metastore, you must combine filters at both levels (for example, using both Include Metastores and Include Catalogs).
Field Description
Account Level Access
  • Enabled (default): Identities are read from the Databricks account, and workspace assignments can be made. This requires the account admin role.
  • Disabled: Identities are not read from the Databricks account. You must provide the Metastore Workspace Pairs field instead.
Concurrency

The maximum number of parallel workers that are used to traverse catalogs and schemas during synchronization. The default value is 4.
Metastore Workspace Pairs

When the Account Level Access option is disabled, provide a list of metastore-workspace pairs.

  • Metastore: The ID of the metastore.
  • Workspaces: A list of workspace deployment names that are associated with the metastore. This can be found at the beginning of your Databricks browser URL, that is, the part before the first dot (for example, adb-xxxxxxxx or dbc-xxxxxxxx). Do not include https:// or the domain suffix.
SQL Warehouses (optional)

A list of SQL warehouses that can be used for synchronization. This field is required to synchronize tags, column masks, and row filters.

  • Warehouse ID: The ID of the SQL warehouse to use for synchronization.
  • Workspace: The workspace for the SQL warehouse. This can be found at the beginning of your Databricks browser URL, that is, the part before the first dot (for example, adb-xxxxxxxx or dbc-xxxxxxxx). Do not include https:// or the domain suffix.
Exclude Catalogs (optional)

A list of catalogs to exclude from synchronization. You can use regular expressions (for example, TEST.*) to match multiple catalogs at once.

Note Exclusion takes priority over inclusion. If a catalog is specified in both the Exclude Catalogs and Include Catalogs fields, it is excluded from synchronization.
Exclude Metastores (optional)

A list of metastores to exclude from synchronization. You can use regular expressions (for example, TEST.*) to match multiple metastores at once.

Note 
  • This field applies only if the Account Level Access option is enabled.
  • Exclusion takes priority over inclusion. If a metastore is specified in both the Exclude Metastores and Include Metastores fields, it is excluded from synchronization.
Exclude Schemas (optional)

A list of schemas to exclude from synchronization. You can use regular expressions (for example, TEST.*) to match multiple schemas at once.

Note Exclusion takes priority over inclusion. If a schema is specified in both the Exclude Schemas and Include Schemas fields, it is excluded from synchronization.
Exclude Tables (optional)

A list of tables to exclude from synchronization. You can use regular expressions (for example, TEST.*) to match multiple tables at once.

Note Exclusion takes priority over inclusion. If a table is specified in both the Exclude Tables and Include Tables fields, it is excluded from synchronization.
Exclude Workspaces (optional)

A list of workspaces to exclude from synchronization. You can use regular expressions (for example, TEST.*) to match multiple workspaces at once.

Note 
  • This field applies only if the Account Level Access option is enabled.
  • Exclusion takes priority over inclusion. If a workspace is specified in both the Exclude Workspaces and Include Workspaces fields, it is excluded from synchronization.
Include Catalogs (optional)

A list of catalogs to include in synchronization. You can use regular expressions (for example, TEST.*) to match multiple catalogs at once.
Include Metastores (optional)

A list of metastores to include in synchronization. You can use regular expressions (for example, TEST.*) to match multiple metastores at once.

Note This field applies only if the Account Level Access option is enabled.
Include Schemas (optional)

A list of schemas to include in synchronization. You can use regular expressions (for example, TEST.*) to match multiple schemas at once.
Include Tables (optional)

A list of tables to include in synchronization. You can use regular expressions (for example, TEST.*) to match multiple tables at once.
Include Workspaces (optional)

A list of workspaces to include in synchronization. You can use regular expressions (for example, TEST.*) to match multiple workspaces at once.

Note This field applies only if the Account Level Access option is enabled.
Additional parameters > Skip TLS Certificate Verification
  • Enabled: The connector does not verify the TLS certificates that are presented by Databricks and its authentication endpoints.
  • Disabled (default): The connector verifies the TLS certificates that are presented by Databricks and its authentication endpoints.
Important Enable this option only as a last resort, in environments with a TLS-inspecting proxy where the proxy's certificate authority (CA) cannot be added to the trust store. Enabling this option disables protection against man-in-the-middle attacks.
Additional parameters > Custom Masking Functions (optional)

A list of fully qualified Unity Catalog function names that can be used as masking methods.

Enter the fully qualified name of the function, separated by dots: my_catalog.my_schema.my_mask.

A custom masking function is used as a masking method only if it meets all of the following criteria:

  • The function exists in Unity Catalog.
  • The function accepts exactly one input parameter.
  • The function returns a value of the same data type that it receives.

The function's comment is shown as its description in Data Access. Beneficiaries of a column mask that uses a custom masking function need the EXECUTE privilege on the function, as well as the USE CATALOG and USE SCHEMA privileges on its parent catalog and schema.

If a function fails these checks, it is skipped with a warning, and the column is masked with NULL instead.
Include Metastore in Grant Name
  • Enabled: The metastore name is included in the access grant name.
  • Disabled (default): The metastore name is not included in the access grant name.

Enable this option if multiple metastores are included in the synchronization, to avoid name clashes.
Tag overwrite key for owners (optional)

An existing Databricks tag key on your data objects that automatically assigns ownership in Data Access.

If a tag on a data object has a key that matches the specified key, the tag's value is set as the owner of the data object in Data Access. If the data object already has an owner, the new value is added alongside the existing one.

Example If you enter the tag key data_steward in this field, and if a table in your data source has the tag data_steward:[email protected], [email protected] is set as the owner of the table in Data Access.
Tag key and value to flag user as a machine (optional)

An existing Databricks tag key and tag value that determine whether an identity is classified as a machine user in Data Access.

Enter the tag key-value pair, separated by a colon: key:value

If a tag on a user has a key-value pair that matches the specified pair, the user is set as a machine user in Data Access, instead of a human user.

Example If you enter the tag key-value pair user_type:service_account in this field, any user in your data source that has the tag user_type:service_account is classified as a machine user in Data Access.

Related topics

Add Databricks to Data Access