Your Spark jobs are now sized from the actual capacity of your cluster's nodes, and start with a set of Spark settings chosen for the Syntasa platform rather than Spark's generic defaults. It is on by default, and every setting it applies can still be overridden from your runtime template.
Default true · Set to false to return to Spark's own defaults · Applies to Kubernetes runtimes only
What it does
Before this, a Spark job on a Kubernetes runtime used the same fixed executor size no matter what machine it landed on. A job on a 64-core node got the same executors as one on a 4-core node, so larger machines went mostly unused unless someone tuned each template by hand.
Two things change when it is on.
Your job is sized from your nodes
The platform reads the real allocatable CPU and memory of the nodes in your runtime and derives the executor shape, driver shape, and executor ceiling from it. Add bigger machines and your jobs use them without editing anything.
Your job starts with better Spark settings
Faster serialization, Arrow-accelerated conversion between Spark and pandas, adaptive query execution with skew-join handling, and dynamic allocation tuned so idle capacity is actually released instead of held for the life of the job.
Turning it on and off
This is a Spark configuration key, not a platform setting or an environment variable. Set it in either place below. It is read each time a job is submitted, so a change takes effect on your next job — nothing needs restarting, and jobs already running are unaffected.
| Global infra-Spark config |
| Applies to every Kubernetes Spark job in the environment. |
| Runtime template Spark config |
| Applies to that one template, and wins over the global value. |
Accepted values
| You set | Result | Notes |
|---|---|---|
| (not set) | on | The default. You do not need to add the key to get the feature. |
| true | on | Case-insensitive; surrounding spaces are ignored. |
| false | off | The rollback switch. Your jobs run on Spark's own defaults. |
| yes / 1 / on / (empty) | off | Anything that isn't a boolean turns the feature off and writes a warning to the runtime service log naming the value you set. |
Why a non-boolean turns it off
Failing closed is deliberate: if you typed something the platform cannot read, it stops guessing and tells you, rather than applying settings you may not have intended. Use exactly true or false.
How your job is sized
Sizing applies to Kubernetes runtimes in cluster mode. The shape is derived from the smallest node in your pool, so a job is never sized for capacity that only some of your nodes have.
Multi-node runtimes
- The driver gets a whole node. It is sized to fill one node so nothing else can be scheduled alongside it, which keeps the executor count honest.
- Executors take the remaining nodes. The ceiling is
(nodes − 1) × executors per node. The node count is your autoscaler maximum where autoscaling is enabled, otherwise your fixed worker count. - Executor cores are capped at 8, even on very large machines — a JVM with many more cores than that has poor garbage-collection behaviour.
- On GPU nodes, you get one executor per GPU device, with cores split evenly between them and the per-task GPU share derived to match.
Single-node runtimes
A one-node runtime runs in local mode: a single pod that gets the whole usable capacity of the machine, with no executors. Executor sizing settings in your template are ignored here, because there are no executors to size.
Sizing is a ceiling, not a reservation
The executor maximum is what your job may grow to, not what it takes at launch. Dynamic allocation starts from zero and adds executors only as real work backs up, then releases them when they go idle. A small job on a large cluster stays small.
Settings applied
Everything below is applied only when the feature is on, and every one of them can be overridden from your runtime template. The values are chosen for Syntasa's Kubernetes runtimes, where there is no external shuffle service and acquiring a new executor can mean waiting for a new machine.
Sizing and capacity
| Setting | Value | What it means for you |
|---|---|---|
| spark.executor.cores | derived | From your node's usable cores, capped at 8. |
| spark.executor.memory | derived | Heap, after the non-heap overhead is set aside. |
| spark.executor.memoryOverhead | derived | Larger for PySpark, which needs room outside the JVM. |
| spark.dynamicAllocation.maxExecutors | derived | Your ceiling. Executor nodes × executors per node. |
| spark.dynamicAllocation.minExecutors | 0 | An idle job holds nothing. |
| spark.kubernetes.allocation.maxPendingPods | derived | Caps how many executor pods can be waiting for a node at once. |
Query performance
| Setting | Value | What it means for you |
|---|---|---|
| spark.serializer | Kryo | Faster and more compact than Java serialization. |
| spark.sql.adaptive.enabled | true | Spark re-plans joins and partition counts using real runtime statistics. |
| spark.sql.adaptive.skewJoin.enabled | true | Splits oversized partitions so one skewed key can't stall a stage. |
| spark.sql.adaptive.advisoryPartitionSizeInBytes | 64MB | Target size when adaptive execution coalesces partitions. |
| spark.sql.optimizer.dynamicPartitionPruning.enabled | true | Skips partitions a join proves it doesn't need. |
| spark.sql.parquet.enableVectorizedReader | true | Batch Parquet reads instead of row-at-a-time. |
| spark.sql.execution.arrow.pyspark.enabled | true | Arrow-accelerated toPandas() and createDataFrame(). |
Capacity release
Kubernetes has no external shuffle service, so releasing an executor destroys the shuffle data it holds. These values decide how long capacity is held before it is given back — and each one is a floor, so an executor holding cached data is kept for the longest applicable window, not the shortest.
| Setting | Value | What it means for you |
|---|---|---|
| spark.dynamicAllocation.executorIdleTimeout | 120s | An executor with nothing to do is released after two minutes. Streaming jobs get 900s so executors survive the gap between triggers. |
| spark.dynamicAllocation.cachedExecutorIdleTimeout | 30m | Executors holding cached data are kept much longer, because you asked for that cache explicitly. Spark's default holds them forever. |
| spark.dynamicAllocation.shuffleTracking.timeout | 120s | How long an executor is kept for shuffle data a later stage might read. If it is released and the data is needed, that stage recomputes. |
| spark.cleaner.periodicGC.interval | 5min | Cleans up shuffle files and broadcast state promptly so long-lived nodes don't fill their disks. |
Reliability
| Setting | Value | What it means for you |
|---|---|---|
| spark.shuffle.io.maxRetries | 6 | More patience with a slow or briefly unreachable peer. |
| spark.shuffle.io.retryWait | 10s | Longer gap between those retries. |
| spark.executor.heartbeatInterval | 20s | A busy executor is less likely to be mistaken for a dead one. |
| spark.speculation | false | Off by design. Duplicate task copies waste capacity on a bounded cluster. |
Overriding a setting
You do not have to choose between the platform's settings and your own. Configuration is layered, and your template is the top layer:
platform defaults < global infra config < your runtime template
Anything you set in your runtime template wins. Set only the keys you want to change; everything you leave alone keeps the platform value.
Override the shape, keep the maths
If you override spark.executor.cores or spark.executor.memory, the platform recalculates the executor ceiling for the shape you actually chose. Ask for executors half the planned size and twice as many now fit per node, so your maximum rises to match — you are not left with a ceiling sized for executors you are no longer using.
If your executors don't fit
Requesting an executor larger than a single node leaves those pods unschedulable — they will wait forever. The platform detects this, leaves the original ceiling alone rather than inventing one for a shape that cannot run, and writes a warning to the runtime service log. If your job never starts any executors, check this first.
What it doesn't touch
- EMR, Dataproc and Azure runtimes. This is Kubernetes-only. Native cloud runtimes keep their existing behaviour.
- Notebook steps. The notebook orchestrator is deliberately excluded from sizing. Its own kernel runs a separate Spark session, and giving the orchestrator a whole-node executor would starve it.
- Your shuffle partition count.
spark.sql.shuffle.partitionsis never set by the platform. A streaming job's saved state is partitioned by it, so changing it would break your streams against their existing checkpoints. - Anything you set yourself. Your template always wins.
Checking what your job got
The definitive answer is always the Spark UI, which shows the configuration exactly as Spark received it after every layer has merged.
Open the Spark UI for your job from the run in the Syntasa console.
Go to the Environment tab and find the Spark Properties table. Everything described on this page appears there with its final value.
Confirm the feature is on by looking for the derived sizing keys —
spark.dynamicAllocation.maxExecutorsandspark.executor.cores. If they are absent, the feature is off for this job.For the sizing decision itself, ask your platform administrator for the
[autosize]line in the runtime service log. It records the node capacity that was read, the shape chosen, and any recalculation triggered by your template overrides.
Common questions
- Do I need to change my runtime templates?
No. It is on by default and derives everything it needs from your cluster. Existing templates keep working, and anything they already set continues to win.
- My job used to hold executors for the whole run and now it doesn't. Why?
That is the intended change. Executors that sit idle are released so other work can use the capacity. If a later stage then needs shuffle data from a released executor, Spark recomputes it. If you would rather trade capacity for avoiding that recompute, raise
spark.dynamicAllocation.shuffleTracking.timeoutin your template.- Will my job cost more?
Generally the opposite. Executors are released when idle rather than held to the end, and the ceiling is bounded by your cluster rather than unbounded. What can rise is peak concurrency, since jobs now use the machines you are already paying for more fully.
- Can I turn it off for one runtime but leave it on everywhere else?
Yes. Set
spark.syntasa.optimizedDefaults=falsein that runtime's template. Template values beat the global setting.- My pandas UDF got slower, or Arrow doesn't seem to be used.
Arrow acceleration needs the matching platform release, which pins the pyarrow, pandas and numpy versions across driver and executor images. If the platform config was applied without those images, Spark falls back silently. Confirm with your administrator that both parts of the release are deployed.
- Does anything change for streaming jobs?
Executors are held for 900 seconds rather than 120 between triggers, so a stream with a multi-minute trigger interval isn't releasing and reacquiring its executors every cycle. Your shuffle partition count is untouched, so existing checkpoints keep working.