Options API

Field mappings

Field name	CLI prop	Description
dataset	ds	dataset name, example: userTable or users or user_file
rundId	rd	run date, must be in format yyyy-MM-dd or for incremental use Hours or Minutes yyyy-MM-dd HH:mm
rundIdEnd	rdEnd	end date for query ranges t_date >= $ and t_date < $, must be in format yyyy-MM-dd or for incremental use Hours or Minutes yyyy-MM-dd HH:mm
passFail	passfaillimit	Limit for passing or failing runs
jobId
onReadOnly	readonly	Do not persist results to the DQ metastore - useful during testing.

Load Options

 

Field name	CLI prop	Description
fullFile	fullfile	use entire file for lookbacks instead of just filequery
fileQuery
header
headerSrc
hasHeader
isParallel
isJson
isMixedJson
isMapsJson
flatten
isMultiLine
sparkprinc
sparkkeytab
jdbcprinc
jdbckeytab
srcpwdmgr
pwdmgr
pguser	pguser
pgpassword	pgpassword
pghost	host
pgport	port
executorcores
isParquet
isAvro
avroSchema
isXml
xmlRowTag
isOrc
dateFmt
timeFmt
datasetSafety
filePath
delimiter
fileLookBack
dbLookBack
connectionURL
userName
password
sqlQuery
connectionProps
zkHost	~~~~	Deprecated
zkPort	~~~~	Deprecated
zkPath	~~~~	Deprecated

Outlier Options

Field name	CLI prop	Description
on	dl	Deep learning.
lookback	dllb	A deep learning lookback. For example, a value 5 for a lookback of 5 days. Note Lookback periods directly correlate to -dllb.
key	dlkey	A comma-delimited deep learning key. comma delim key ex: symbol,date
dateField
bin
includes	dlinc	A deep learning col limit. For example, open,close,high,volume.
excludes	dlexc	A deep learning col exclusion. For example, open,close,high,volume.
categorical
by
limit
historyLimit
minhistory	dlminhist	An automatically generated flag based on the outlier lookback setting -dllb that defines the minimum number of days before DQ flags data as potential outliers. -dlminhistensures that the number of days in the algorithm is relative to the total scope of the lookback period. It is not recommended that you override this flag from the command line. Note Lookback periods do not directly correlate to -dlminhist.
score

FPG Options

Field name	CLI prop	Description
on	fpgon	pattern mining
lookback	fpglb	lookback interval for pattern mining. Ex: -fpglb 5
key	fpgkey	natural key for pattern mining activity
dateField	fpgdc	date column for pattern mining. Ex: -fpgdc date_col
lowFreq		Deprecated
includes	fpginc	pattern mining is expensive use this input to limit the observed cols
excludes	fpgexc	pattern mining is expensive use this input to limit the observed cols
timeBin	fpgtbin	time bin for pattern mining. Ex: -fpgtbin DAY
score	fpgscore	score for pattern mining records
minSupport	fpgsupport
confidence	fpgconfidence

Dupe Options

Field name	CLI prop	Description
on	dupe	duplicate record detection
includes	dupeinc	duplicate record detection, column inclusion list
excludes	dupeexc	duplicate record detection, column exclusion list
depth
lowerBound	dupelb	duplicate lower bounds on percent match default [85]
upperBound
blocksize
useCache
checkHeader
exactMatch
ignoreCase	dupenocase	duplicate record detection, column exclusion list
score	dupescore
limit	dupelimit	Limit for dupe rows stored

Profile Options

 

 

Field name	CLI prop	Description
on
includes
excludes
dataShapeOn
statsOn
correlationOn
histogramOn
cardinalityOn
dataShapeColsInc
dataShapeColsExc

Source Options

Field name	CLI prop	Description
on
includes
excludes
key
fileQuery
map
score
datasetSrc
driverSrc
userNameSrc
passwordSrc
connectionURLSrc
sqlQuerySrc
connectionPropsSrc

Rule Options

Field name	CLI prop	Description
on
rulesOnly
semantic

ColMatch Options

Field name	CLI prop	Description
colMatchParallelProcesses
colMatchDurationMins
colMatchBatchSize
connectionList

Spark Options

Field name	CLI prop	Description
numExecutors
executorMemory
driverMemory
executorCores
master
jars
libs
driver

Misc Options

Field name	CLI prop	Description
obslimit
nullValue