A notebook is an interactive document that blends live code, narrative text, and the results of code execution (including visualizations like charts and tables) into a single, cohesive document. The most well-known example is the Jupyter Notebook, a free and open-source application. Other key terms are PySpark and Databricks:
- PySpark is the Python library that provides the API for interacting with the Apache Spark big data engine.
- Databricks is a unified analytics platform that provides a hosted, optimized environment for running Spark (and thus PySpark) on large-scale clusters, often using Jupyter-like notebooks.
In the context of Data Quality & Observability Classic, notebooks provide an alternate method to process data. Advantages of using notebooks with DQ include:
- Greater flexibility to handle specific actions.
- Leverage existing data ingestion pipelines.
- Reuse existing orchestration and cluster resources.
- Access data without a JDBC connection.
- Scale automation.
Job results are stored in the DQ metastore and accessible from the DQ user interface. However, one common use case for notebooks is to review job results for specific situations–-and handle them–-before they're stored in the DQ metastore.