Using a 10/1 ratio of RAM to Executors is often a good rule of thumb, another and more simple option is to turn on dynamic.allocation and allow the resources to be provided as needed on demand.
requires that you have a maximum of 250 columns or less.
Limit Columns
In most cases, there is a large number of columns that go unused by the business or columns that don't require checking. One of the most efficient things you can do is limit the cols using the below cmds. As a best practice strongly recommends using less than 80 columns per dataset.
-q "select colA, colB, colC, datCol, colD from table"
// vs
-q "select * from * from table"
How to limit columns when using a file
-fq "select colA, colB, colC from dataset"
// file query using keyword dataset
JDBC vs Local Data
Co-Located data (local data)
It is always a good performance practice to co-locate data and processing. That doesn't mean that your tech organization chooses to do this in it's architecture and design which is why accounts for both. If the data is located on the cluster that is doing the processing use options like -hive for non JDBC and native file access. Skip tuning for JDBC as moving data to the cluster first will routinely reduce 50 percent of the performance bottleneck.
Set DriverMemory
add more memory to the driver node as it will be responsible for the initial landing of data.
--driver-memory 7g
Add Parallel JDBC
Limit Features, Turn Flags Off
-corroff //only losing visuals, 5% speed gain
-histoff //only losing visuals, 4% speed gain
-hootonly //speeds up 1% based on less logging
-readonly //remove owl webapp read writes, 1% gain
-datashapeoff //removes Shape Detection 3% speed gain
Real World Scenario
Nine million rows with 46 columns on a daily basis for just 1 dataset. The data lives in Greenplum and we want to process it on a cluster platform where runs. The first run results in a 12 minute runtime. While acceptable it's not ideal, here is what you should do:
Add Parallel JDBC for faster network.
Limit columns to the 18 that are of use in the downstream processing.
Turn off unneeded features.
Find out of the job is memory bound or CPU bound.
By setting the below configs this same job ran in six minutes.
