Create a DQ job

This page provides an overview of the steps involved in the creation of a DQ job. Select from one of the options below depending on whether you are using Pullup or Pushdown mode.

  • Pullup
  • Pushdown

Create a Pullup job

Step Action Description
1 Select a data source

Your data source is where your table, file, or view upon which your DQ job runs exists.
2 Select a scanning method Determine whether to scan part of your table, file, or view, a sample of it, or its entirety.
3 Select your columns

Select the columns to include in your DQ job. Optionally assign Key columns and Link IDs. Link IDs are required for archiving break records.

Note This step is optional for Full Scans.
4 Select your rows

Apply filters and limits to your rows for enhanced scanning.

Note This step is optional for Full Scans.
5 Add transforms Modify or create new columns to meet specific formatting requirements at the Spark level.
6 Create a mapping Validate and link source and target column relationships between two data sources.
7 Add DQ layers Customize your DQ job to gain more insights into the health of your data.
8 Review the job sizing Review and modify details about your agent resources.
9 Review your DQ job Review your query before you submit your DQ job.
10 Run your DQ job Submit your DQ job for processing.

Create a Pushdown job

Step Action Description
1 Select a data source

Your data source is where your table, file, or view upon which your DQ job runs exists.
2 Select a scanning method Determine whether to scan part of your table, file, or view, a sample of it, or its entirety.
3 Select your columns

Select the columns to include in your DQ job. Optionally assign Key columns and Link IDs. Link IDs are required for archiving break records.

Note This step is optional for Full Scans.
4 Select your rows

Apply filters and limits to your rows for enhanced scanning.

Note This step is optional for Full Scans.
5 Add DQ layers Customize your DQ job to gain more insights into the health of your data.
6 Review your DQ job Review your query before you submit your DQ job.
7 Run your DQ job Submit your DQ job for processing.