Hardware Usage Report
What is a Hardware Usage Report?
The Hardware Usage report provides a view of the Collibra DQ jobs that have consumed the most hardware.
You can generate a Hardware Usage report from the Reports page.
Steps
-
Sign in to the Collibra DQ instance and click the
Reports icon in the left navigation pane.
The Reports page opens. -
From the Reports page, click the Hardware Usage Report link.
- For x-axis, select one of the following options from the dropdown:
- num_executors
- executor_memory
- total_memory
- total_cores
- executor_cores
- avg_row_count
- dataset
-
For y-axis, select one or more of the following options:
- num_executors
- executor_memory
- total_memory
- total_cores
- executor_cores
- avg_row_count
- dataset
-
For the type of chart you want to display, select one of the following options from the drop-down:
- Line
- Area
- Column
- Scatter
-
Navigate the pages of your report by clicking the Previous and Next pagination buttons, located bottom-right of the columns.
Sample SQL query
You can use the following sample SQL query for a Hardware Usage report.
with most_current_dataset_scan as ( select dataset, avg(rc)::Int as avg_row_count from dataset_scan group by dataset )
select opt.dataset, num_executors, executor_cores, executor_memory, (executor_cores * num_executors) total_cores,
(NULLIF(regexp_replace(executor_memory, '\D', '', 'g'), '')::numeric * num_executors) as total_memory,
ds.avg_row_count
from opt_spark opt, most_current_dataset_scan ds
where opt.dataset = ds.dataset
order by total_cores desc limit 75