SQL-Based Rules
Depending on the complexity, you can choose from short-form or long-form rules.
Simple
Just the condition (short-form). For example, using the column email_address. This runs against the dataframe and uses Spark SQL syntax. Simple rules can be thought of as everything after the where clause.
See the Simple section for more details.
Freeform
Where 'Simple' rules just use the condition, 'Freeform' rules use the complete SQL statement. When more complex SQL is required, you can express more with Freeform including joins and window statements.
See the Freeform section for more details.
Native
Native rules use the SQL dialect of the underlying connection and database. Files are not eligible for native SQL rules. This is ideal if you want to use pushdown profiling and you want to use existing SQL logic. When coupled with pushdown profiling, you can achieve a very minimal infrastructure footprint.
See the Native section for more details.
Stat rules
Write rules against meta data and profiling stats. Complex counts and ratios can be referenced with simple syntax.
See the Stat Rules section for more details.
Data Type rules
See the Data Type Rules section for more details.