Kubernetes Spark jobs and notebooks used to run on one-size-fits-nothing defaults unless you hand-tuned every process. The platform now reads the actual hardware, sizes the job for you, and applies the tuning most teams end up adding by hand — and your own settings still win everywhere.
The short version
- Spark jobs on Kubernetes runtimes are sized from the nodes they run on. Executor cores, memory, and the executor ceiling are computed from real node capacity instead of Spark's 1-core / 1 GB defaults. See the arithmetic, with examples.
- Jobs also get a full set of engine defaults — serialization, adaptive queries, Arrow, dynamic-allocation timeouts that actually release idle executors. The complete list is below.
- Anything you already tuned is untouched. Platform values sit underneath your global Spark config and your runtime templates. If you set a value, yours wins. Always.
- Notebooks got a clear model. A notebook without an attached runtime is a fast, predictable exploration tier; attaching a runtime is how you scale.
Why we did this
Before this release, a Spark job on a Kubernetes runtime that wasn't hand-tuned ran with Spark's own defaults: 1-core, 1 GB executors regardless of node size, an unlimited executor count that could pile up Pending pods, and idle executors that were never released once they held shuffle data. A 32 GB single-node runtime would happily run its whole job in a 1 GB heap.
Tuning all of that per process is expert work that most teams shouldn't have to do. So the platform now does the arithmetic — and this article walks through exactly what it sets and how.
How the platform sizes your job
When a job is submitted to a Kubernetes runtime, the platform reads the runtime's live node capacity and computes a plan in four steps:
- Trim the node to its usable share. Roughly 85% of memory and all-but-one core are usable for Spark (a bit more on nodes that already carve out system reserves) — the rest is headroom for the kubelet and system daemons.
- Decide executors per node. Executors are capped at 8 cores and a 64 GiB container each. A node bigger than one cap is split into several executors; on GPU nodes, never more executors than GPU devices.
- Shape each executor. The per-executor container splits into JVM heap plus memory overhead (heap = container ÷ 1.10 for Scala, ÷ 1.20 for Python — Python needs more room outside the JVM for its workers). On GPU nodes, overhead is floored at 2 GiB for the GPU libraries' native memory.
-
Bound the job.
maxExecutors = (nodes − 1) × executors-per-node— one node's worth stays free for the driver — and the Pending-pod queue gets a small finite cap instead of “unlimited.”
Worked examples
Example 1 — a typical mid-size runtime10 nodes × 16 cores / 64 GiB · Scala job |
|---|
|
| → spark.executor.cores=7 · spark.executor.memory≈24.7g · memoryOverhead≈2.4g · maxExecutors=18 — versus 1-core / 1 GB / unlimited before. |
Example 2 — a small runtime, Python job5 nodes × 4 cores / 32 GiB · PySpark |
|---|
|
| → spark.executor.cores=3 · spark.executor.memory≈22.6g · memoryOverhead≈4.5g · maxExecutors=4 |
Example 3 — a single-node runtime1 node × 4 cores / 32 GiB |
|---|
|
| → one pod, whole node, spark.driver.memory≈23.1g (Scala) — versus a 1 GB heap on the same machine before. |
Every job's log records the plan it computed and applied (look for [autosize] lines), so you can always see where a number came from.
Every default you get, in one table
This is the complete set. Scope tells you what controls each row: gated rows apply to Kubernetes jobs when spark.syntasa.optimizedDefaults=true (the default) and vanish when it's false; all runtimes rows apply everywhere, switch or no switch; unchanged rows were already set before this release and are listed so you have the full picture. “Before” is the value that was effectively in force previously (Spark's own default where we never set one).
| Setting | Before | Now | Scope | |
| Computed per job from node capacity | spark.executor.cores |
1 | node-fitted (≤ 8) | gated |
spark.executor.memory |
1g | node-fitted (≤ 64 GiB container) | gated | |
spark.executor.memoryOverhead |
10% / 384m floor | computed; ≥ 2g on GPU | gated | |
spark.dynamicAllocation.maxExecutors |
unlimited | (nodes − 1) × per-node fit | gated | |
spark.dynamicAllocation.minExecutors |
0 | 0 | gated | |
spark.kubernetes.allocation.maxPendingPods |
unlimited | small finite cap | gated | |
spark.driver.memory (single-node local) |
1g | 85% / 70% of node usable | gated | |
| Dynamic allocation — idle executors get released | …shuffleTracking.timeout |
infinite | 120s | gated |
…executorIdleTimeout |
60s | 120s (900s streaming) | gated | |
…cachedExecutorIdleTimeout |
infinite | 30m | gated | |
spark.kubernetes.allocation.batch.size |
5 | 10 | gated | |
spark.kubernetes.allocation.batch.delay |
10–60s | 1s | gated | |
…dynamicAllocation.enabled |
true | true | unchanged | |
…shuffleTracking.enabled |
true | true | unchanged | |
spark.shuffle.service.enabled |
false | false (pinned) | gated | |
| Serialization & Python data exchange | spark.serializer |
Java | Kryo | gated |
spark.kryoserializer.buffer / .max
|
64k / 64m | 512k / 512m | gated | |
spark.sql.execution.arrow.pyspark.enabled (+ fallback) |
false | true | gated | |
spark.sql.execution.arrow.maxRecordsPerBatch |
10000 | 10000 (pinned) | gated | |
| Query engine |
spark.sql.adaptive.enabled + skewJoin
|
varies | true | gated |
…adaptive.coalescePartitions.parallelismFirst |
true | false — target 64MB partitions | gated | |
…adaptive.advisoryPartitionSizeInBytes |
64MB | 64MB (pinned) | gated | |
…dynamicPartitionPruning / parquet.enableVectorizedReader
|
true | true (pinned) | gated | |
spark.speculation |
false | false (pinned) | gated | |
| Reliability |
spark.shuffle.io.maxRetries / retryWait
|
3 / 5s | 6 / 10s | gated |
spark.executor.heartbeatInterval |
10s | 20s | gated | |
| Notebook jobs only | spark.databricks.delta.vacuum.parallelDelete.enabled |
true | true | gated |
| Changed on every runtime type (incl. EMR & Dataproc) | spark.sql.autoBroadcastJoinThreshold |
30MB | 10MB | all runtimes |
spark.sql.adaptive.autoBroadcastJoinThreshold |
— | 30MB | all runtimes | |
spark.network.timeout |
600s | 300s | all runtimes | |
spark.shuffle.io.connectionTimeout |
= network.timeout | 120s | gated |
Two rows worth a sentence each. Shuffle tracking (shuffleTracking.enabled=true) was already on — it's how dynamic allocation works on Kubernetes, where there is no external shuffle service; what changed is its timeout, which finally makes it release executors. And the broadcast threshold split in two: the static threshold (based on often-stale table statistics) dropped to 10 MB to stop driver out-of-memory failures, while the adaptive threshold (based on sizes measured at runtime) holds the old 30 MB — so joins that really are small still broadcast.
Your settings still win
Every gated value enters at the lowest layer of the config stack: platform < global < runtime template. A template that sets spark.executor.memory, a timeout, or anything else keeps its value exactly as before — the plan even recomputes its executor ceiling around your shape rather than fighting it.
The rollback switch
All the gated rows above hang off one Spark config key:
Config key
spark.syntasa.optimizedDefaults — true (default) applies everything above. false restores the previous release's behavior for that scope, including the long-standing notebook-job safeguards that predate this work.
| Where you set it | What it affects | Wins over |
|---|---|---|
| Global Spark config (Infrastructure page) | Every Kubernetes job, notebook job, and attached notebook session in the environment | Platform defaults |
| Runtime template (template's Spark config) | Jobs and sessions on that runtime only | The global setting |
New kernels and jobs pick up a change on their next launch — no service restarts needed. The all runtimes rows aren't gated; override those keys directly if a workload needs the old values.
Notebooks: attachment decides the mode
Notebook behavior now follows one simple rule: whether a runtime is attached decides how your kernel runs.
| Local |
No runtime attachedThe exploration tier. Your kernel runs on the shared notebook pool with a fixed, guaranteed slice — no executor pods, no waiting, no surprises from noisy neighbors. 1 core · 8 GB pod · heap 2 GB (Python) / 5 GB (Scala) |
|---|---|
Single-node runtime attachedThe kernel gets the whole node to itself and runs Spark in-process — the fastest way to use one big machine. | |
| Distributed |
Multi-node runtime attachedReal cluster Spark: executors on the runtime's nodes, sized by the same arithmetic as jobs, with a finite executor ceiling and idle executors released. executors ≤ (nodes − 1) × per-node fit |
Notebook job (scheduled)The coordinator that drives your notebook now takes a small fixed slice instead of a whole node, leaving the rest for the kernel doing the actual work. coordinator 1 core / 2 GB · kernel gets the rest |
What changed for un-attached notebooks — and why
Previously, a notebook with no runtime attached could quietly launch up to five executor pods on the shared pool, while its own pod sat far above its memory guarantee — making it the first thing evicted when the pool got busy. Now the un-attached kernel is a Guaranteed-priority pod with a fixed budget: it's the last thing evicted, its performance is consistent regardless of neighbors, and three kernels pack cleanly on each pool node.
The trade: un-attached notebooks no longer spin up executors at all. If a notebook needs more than one core and 8 GB, that's the signal to attach a runtime — which is also where you choose the hardware that fits the work.
The Python and Scala heaps differ on purpose. In a Python notebook your data lives in pandas, outside the JVM — so the JVM gets 2 GB and pandas gets the rest. In a Scala notebook your data lives inside the JVM — so it gets 5 GB.
GPU notebooks
GPU kernels are exempt from the fixed budget: the kernel pod owns the GPU and gets the whole node's memory, since GPU work is exactly the case where a small fixed slice would hurt. On single-node GPU runtimes running a notebook job, the coordinator now stays off the GPUs entirely — the devices and the GPU libraries belong to your kernel, not the plumbing around it.
Heads-up
Un-attached notebook sizing is not controlled by the optimizedDefaults switch — it's the shape of the shared pool itself, and there's no runtime template in the picture to read the switch from. Attached sessions and notebook jobs respect the switch like everything else.
Questions you might have
My job's executor sizes look different after the upgrade. Is something wrong?
That's the feature working. Jobs that never set explicit executor sizes now get node-fitted executors instead of 1-core / 1 GB ones — most jobs get faster. If a specific job regresses, set your preferred values in its runtime template (they win), or set spark.syntasa.optimizedDefaults=false on that template.
Why one big executor per node instead of the usual 4–5 cores per executor?
The classic 4–5-core advice comes from HDFS client throughput limits, which don't apply when reading from S3, GCS, or ABFS. Fat executors also mean releasing one executor empties a whole node — so autoscaled runtimes can actually shrink. Very large nodes still split (nothing exceeds 8 cores / 64 GiB per executor).
My un-attached notebook used to run heavier workloads. What now?
Attach a runtime. A single-node runtime gives your kernel a whole machine; a multi-node runtime gives it real executors with proper sizing. The shared pool is deliberately a fixed-size exploration tier now — that's what makes it fast and predictable for everyone.
How do I turn all of this off?
Set spark.syntasa.optimizedDefaults=false in the global Spark config (whole environment) or on a runtime template (just that runtime). New sessions and jobs pick it up on launch. The all-runtimes rows (broadcast threshold, network timeout) aren't gated — override those keys directly if you need the old values.
Does any of this change EMR or Dataproc jobs?
The sizing engine and the switch apply to Kubernetes runtimes only. EMR and Dataproc pick up just the rows marked “all runtimes” in the table — nothing about their executor sizing or allocation behavior changes.
Will the platform overwrite the tuning we spent months on?
No. Platform values load first and lose to everything: your global config overrides them, and your runtime templates override both. The only settings written unconditionally are the all-runtimes rows, and even those yield to an explicit value in your config layers.