Pullup job sizing recommendations

To ensure optimal performance, efficient resource utilization, and successful job execution, you can fine-tune the job resource settings from the global job limits page. This is especially important when dealing with exceptionally large sets of data.

The following table shows recommendations on how to scale your job in different circumstances.

High-level resource targets

Depending on your data size, your job requires a total target amount of RAM and CPU cores to run efficiently. The following table outlines these high-level capacity targets.

Total rows Total columns Total cores Total RAM
100K 51 2 3 GB
1M 51 3 6 GB
10M 51 12 24 GB
20M 51 21 33 GB
50M 51 45 72 GB

Calculating total resources

The global limits page does not accept "total" inputs, as described in the previous table. Instead, Spark distributes these total resources between a single driver node and multiple executor nodes. The underlying engine allocates resources based on these two formulas:

  • Total cores = Driver cores + (Number of executors × Executor cores)
  • Total RAM = Driver memory + (Number of executors × Executor memory)

Example For a table with 10M rows, you can achieve a target of 12 total cores and 24 GB total RAM by configuring 1 driver with 4 cores and 4 GB RAM, plus 4 executors that each have 2 cores and 5 GB RAM.

Recommended settings on the global limits page

To achieve the total targets described in the previous table, use the following baseline settings for the specific fields on the global limits page.

Note The following values are intended for illustrative purposes only.

Total rows Total columns Maximum number of driver cores Maximum driver memory Maximum number of executors Maximum executor cores Maximum executor memory
100K 50 1 2 GB 2 2 4 GB
1M 50 1 2 GB 1 2 4 GB
10M 50 1 4 GB 4 2 16 GB

Additional settings

For the remaining fields on the global limits page, use the following guidelines unless your specific workload dictates otherwise:

  • Maximum worker cores: Set this equal to or slightly higher than your executor cores.
  • Maximum worker memory: Set this equal to or slightly higher than your executor memory to allow for system overhead.

Running jobs with Parallel JDBC

Parallel JDBC splits the data-load stage across partitions so executors pull data in parallel. It delivers the biggest gains on large tables, where the load stage is the bottleneck. On small tables the load stage is already fast, so there is little to gain.

Example A: 10M rows

Both configurations use four executors with two cores each. Enabling Parallel JDBC requires more memory per executor (12 GB vs. 6 GB) and more driver memory (4 GB vs. 1 GB) to support parallel data loading across 50 partitions. The additional memory allocation allows executors to pull data simultaneously rather than sequentially, which cuts run time by roughly 70%, from 25 minutes 50 seconds down to 7 minutes 32 seconds.

Global limits setting Without Parallel JDBC With Parallel JDBC
Executors 4 4
Executor cores 2 2
Executor memory 6 GB 12 GB
Driver memory 1 GB 4 GB
Number of partitions 46 50
Run time 25m 50s 7m 32s

Example B: 1M rows

Both configurations are identical in resource allocation and partition count. For a small table with only 1M rows, the data load stage completes quickly regardless of whether partitions are pulled in parallel or sequentially. As a result, Parallel JDBC provides no meaningful benefit at this scale. The difference in run time is less than one minute.

Global limits setting Without Parallel JDBC With Parallel JDBC
Executors 2 2
Executor cores 1 1
Executor memory 2 GB 2 GB
Driver memory 2 GB 2 GB
Number of partitions 5 5
Run time 15m 50s 15m 8s

When to use Parallel JDBC

Use Parallel JDBC for large tables (roughly 10M rows or more), where it can cut run time dramatically. For small tables (100K–1M rows) the gain is marginal, so it is not worth the extra configuration. Set the number of partitions to match the data size. For example, the benchmark runs used approximately 50 partitions at 10M rows, scaling up to 200 at 50M rows.

What's next