Import Data Quality jobs (in preview)

You can import Data Quality jobs to promote jobs created in a non-production environment or any Data Quality & Observability (self-hosted) environment into a production Data Quality & Observability (cloud) environment. The import wizard guides you through uploading an export file, mapping connections, resolving conflicts, and reviewing the configuration before you submit.

Prerequisites

  • The import feature flag is enabled.

  • You have a JSON export file from the source environment.

  • You have the required permissions to access Settings > Data Quality.

Steps

  1. Go to Settings > Data Quality > Import.

    The Import page opens with a list of prior imports.

  2. Click Import to open the import wizard.

    The Upload configuration file step of the wizard opens.

  3. Upload your DQ export file in JSON format.

    The file is validated after upload. The maximum file size is 17 MB by default, but can be changed on the Limits page. If the file is invalid, an error state is shown and you cannot proceed.

  4. Click Next.

    The Connection mapping step of the wizard opens.

  5. Map each Source connection in the export file to a Target UDQ connection. Target connections must share the same datasource type and processing type as the source connection. You must map all connections before proceeding.

  6. Click Next.

    The Validate step of the wizard opens.

  7. Review each mapped connection for conflicts. You must resolve all conflicts shown on the Blocked jobs tab before proceeding to the next connection. For connections with conflicts:

    1. Select a Conflict resolution option:

      1. Skip - doesn’t import the job.

      2. Overwrite - replaces the existing Data Quality & Observability (cloud) job with the imported job.

      3. Rename - appends a suffix to the imported job name.

    2. Enter the Suffix to append to the modified jobs. This is required if Rename is selected as the Conflict resolution.

    3. Select the Database that applies to every job imported for this connection.

    4. If necessary, select Enable descriptive statistics for all migrated jobs. When selected, the descriptive statistics in profiling are shown for all jobs.

      Note This may expose sensitive values, such as a minimum or maximum values.

    5. If you have multiple source to target connections, then you must validate each of them separately. Click Next to move to each connection. When you finish the last connection, the button changes to Continue to review.

  8. Click Next or Continue to review.

    The Review step of the wizard opens.

  9. Review the import summary.

    The summary shows, for each connection: the number of jobs, monitors, and notifications to be imported, the database product name, and the processing type.

  10. Click Submit to start the import.

    The wizard closes on success. If the import fails, an error notification is displayed and the wizard remains open. The import is logged in the import history table on the Import page.

  11. Ensure that the imported jobs have been run on the Job management page. To identify jobs that were imported but not yet run, sort the list by the Last Execution Date column. You must manually run unscheduled jobs, while scheduled jobs will run automatically according to their schedule.

View jobs

On the Import page, select More icon, then View jobs. The View jobs dialog box opens and shows additional details about the import's jobs, such as the source, target, result, and target job ID.

Download report

On the Import page, select More icon, then Download report. The import report downloads in JSON format.

Note The file must be downloaded within 8 hours of the import.

Handling monitor warnings during job export/import

When exporting a job from one instance and importing it into another, the number of active monitors may differ between the two instances if there are warnings associated with certain monitors during the export or import process. Monitors that generate warnings may not be imported and therefore will not appear in the destination instance.

Example A job has 14 active monitors in the source instance, and two of those monitors are rules without a primary column. If the MANDATORY_PRIMARY_RULE_COLUMN flag is enabled, exporting and importing the job may generate the warning: "Primary Column is required when MANDATORY_PRIMARY_RULE_COLUMN is turned on". As a result, those monitors may not be imported and the destination instance will have fewer than the original 14 active monitors.

To import all monitors successfully, you must do one of the following:

  • Turn off the MANDATORY_PRIMARY_RULE_COLUMN flag before exporting and importing.

  • Add a primary column to the affected rules, then export and import again.

Related topics

About DQ job import