Collibra DQ Rules
In this section, you can learn how to work with Rules in Notebooks written in Scala.
Instantiation
Code | Description |
---|---|
val = new Rule() | Instantiating a Rule object |
rule.setDataset(<DATASET>) | Adding the name of the dataset |
rule.setRuleNm(<RULE_NAME>) | Adding the name of the given rule |
rule.setRuleValue(<RULE_EXPRESSION>) | Setting the simple RULE_EXPRESSION |
rule.setRuleType(<RULE_TYPE>) | Setting the rule type |
rule.setPerc(<RULE_PERCENTAGE>) | Setting the percentage |
rule.setPoints(<RULE_POINT>) | Setting how many points will be deducted from total for each percentage |
rule.setIsActive(<RULE_IS_ACTIVE>) |
Making rule active/inactive Possible values: ACTIVE: 1 / true INACTive: 0 / false |
rule.setUserNm(<RULE_OWNER_USER_NAME>) | Adding the owner |
Rule Types
Requirements
Required imports
Copy
import com.owl.core.Owl
import com.owl.core.util.OwlUtils
import com.owl.common.options.OwlOptions
import com.owl.common.{Props, Utils}
import com.owl.common.domain2.Rule
import org.apache.spark.sql.funtions._
SparkSession initialization
Databricks and other notebook execution frameworks are working with managed/shared Spark sessions, therefore, Collibra DQ recommend using this code snippet in your notebook to initialize the current Spark session properly.
Copy
// ----- Init Spark ----- //
def sparkInit(): Unit = {
sparkSession = SparkSession.builder
.master("local")
.appName("test")
.getOrCreate()
}
Warning Do not call spark.stop at any of your notebooks. Otherwise, the execution engine will exit immediately.
Important Make sure OwlContext is already created before using any method from OwlUtils.