Most notebook work is fine on the default kernel — you don't have to attach anything to start. Attach a runtime when you outgrow the default: for example, when you need a bigger driver, more executors, GPU access, or a runtime that has been pre-configured with specific Spark settings and dependencies. This article will walks through the runtime toolbar at the top of every notebook, the two attachment flows, and what happens when you click each button.
What is a runtime template?
A runtime template is an admin-defined recipe for a Kubernetes Spark cluster. It specifies the container image the cluster will run, the instance type and count (or autoscale range), an idle timeout, and whether the cluster is dedicated to a single notebook or can be shared across many. You don't author runtime templates as a notebook user — you pick from the templates your admin has made available to you.
A runtime instance is a live Kubernetes Spark cluster spawned from a template. A dedicated template can have many instances (one per notebook that started one); a shared template typically has just one instance that several notebooks attach to.
The default kernel — what you get without attaching anything
When you open a notebook in a workspace and don't touch the runtime toolbar, the platform gives you a default kernel:
- Driver runs on the workspace's default Jupyter compute pool. The driver pod size is fixed.
- Spark gets a maximum of 5 executors. The count is fixed; you cannot change it from the notebook. The default kernel does not autoscale — only runtime templates can.
- There is also a default GPU compute pool for GPU-enabled kernels — Maximum of 5 executors with 1 gpu for each executor.
This is the right environment for quick interactive testing, exploration, and anything that doesn't need scale. You skip the wait of provisioning a Kubernetes cluster, and you avoid the cost of running one. For most work that fits, the default is the simplest choice.
Terminology — what's a "compute pool" and a "kernel pod"?
When this article talks about a "compute pool," it means the underlying Kubernetes node pool the platform schedules pods onto. The "kernel pod" is the pod that hosts your notebook's kernel and Spark driver. Attaching a runtime moves your kernel pod from the default compute pool to the runtime's compute pool - configured by the runtime template. See the Glossary for short definitions of all of these.
Why attach a runtime?
Attach a runtime when the default isn't enough:
- Bigger driver. The runtime template's container is sized by an admin. If the default driver pod can't hold your workload, attach to a runtime template that gives you more memory and CPU.
- More executors. Runtime templates can be configured for a specific executor count or an autoscale range. Either way you get more parallelism than the fixed five-executor default.
- GPUs. If your work needs GPU acceleration on the executors, attach to a GPU runtime template.
- Pre-configured Spark and dependencies. A runtime template can carry its own init script, JAR list, and Spark configuration that activate when the runtime is attached. See Init Scripts & Dependencies for the layered model and how to configure these at the global, runtime, and notebook level.
The notebook's functionality is the same with or without an attached runtime. What changes is scale, isolation, and the configuration the runtime brings with it.
What if no template fits?
Runtime templates are admin-defined and admin-controlled. There is no notebook-side override — you cannot bump up the driver memory or executor count from a cell. If none of the templates available to you provide enough resources for your workload, the right path is to ask your admin to create a new template (or grant you access to an existing one that does).
When asking, give your admin the specifics: how much driver memory you need, how many executors and at what instance type, whether you need GPUs, and whether the cluster should be dedicated or shared. Note that the template tooltip in the dropdown shows instance type and count but does not show the per-pod memory directly — you can map instance type to memory using the cloud provider's instance-size table, or attach to the template once and check the values in the Spark UI's Environment tab.
The runtime toolbar
The runtime toolbar sits at the top of every notebook. It has four parts, left to right:
- Template dropdown — pick a runtime template.
- Instance dropdown — pick an existing instance to attach to, or create a new one. Hidden until a template is selected.
- Status indicator — a colored dot plus a short text label showing the current state.
- Action button — labeled Start, Attach, Stop, or Detach depending on the state.
Template dropdown
Opening the template dropdown lists every runtime template you have access to. Templates you do not have access to do not appear in the list — visibility is admin-managed. Only Kubernetes Spark templates show up here; YARN-based runtimes (EMR, Dataproc) are not supported by the toolbar.
Hover over a template to see a tooltip with its key metadata: whether the cluster is shared, the instance type, the instance count or autoscale range, and the idle timeout. Use the tooltip to pick the template that matches the work you need to do.
Instance dropdown
Once a template is picked, the instance dropdown appears. It always starts with + Create New Instance, followed by any existing instances of this template that are running or starting up. Each entry shows the instance name and current status — for example, spark-a1b2 (Running).
On a shared template, the + Create New Instance entry is greyed out when there is already an active instance — shared templates are designed to attach many notebooks to a single cluster, not to spawn a new one each time. The toolbar will auto-select the active shared instance for you, so the next click is Attach.
Action button
The action button label depends on what is currently selected and what state it is in:
| Label | When it appears | What it does |
|---|---|---|
| Start | An instance is selected (or + Create New) and not yet running | Creates the instance, waits for it to come up, and attaches your kernel automatically when it is ready |
| Attach | An existing instance is selected and is already Running | Attaches your kernel to that instance — restarts (and reschedules) the kernel pod onto the runtime's node pool |
| Stop | Your kernel is currently attached to a dedicated cluster | Asks for confirmation, then terminates the cluster and returns your kernel to the default node pool |
| Detach | Your kernel is currently attached to a shared cluster | Asks for confirmation, then unbinds your kernel from the cluster (the cluster keeps running for other notebooks) and returns your kernel to the default node pool |
Status indicator
The colored dot to the left of the action button reflects the current state of the runtime.
| State | Color | Meaning |
|---|---|---|
| (idle) | — | No template selected, or no instance picked. Dot hidden. |
| Creating | Amber | Cluster is being created. Be patient — first-time provisioning can take a few minutes. |
| Starting | Amber | Cluster is provisioning further (e.g., pulling images, scaling nodes). |
| Ready / Not attached | Blue | Instance is Running but your kernel is not attached to it yet. |
| Attaching | Amber | Your kernel is moving onto the runtime's node pool. |
| Connected | Green | Your kernel is on the runtime's node pool and ready to use. |
| Detaching | Amber | Your kernel is moving back to the default node pool. |
| Stopping | Amber | Cluster is being terminated. |
| Error | Red | Something failed. Hover the dot to see the message; see Errors and recovery below. |
Two flows: dedicated and shared
There are two ways to use a runtime template, and the toolbar UX is slightly different for each. The template's shared flag (visible in the template tooltip as Shared cluster: Yes / No) determines which flow applies.
Dedicated cluster (one cluster per notebook)
With a dedicated template, every notebook that wants to use it spins up its own cluster. The cluster is yours alone — no other notebook attaches to it.
- Pick the template from the template dropdown.
- Leave + Create New Instance selected (the default), or pick an existing instance you started earlier.
- Click Start. The status indicator shows Creating → Starting → Connected as the cluster comes up and your kernel attaches.
- Run cells. Your kernel is on the runtime's node pool, your driver has the runtime's resources, your executors are sized by the template.
- When you're done, click Stop. Confirm the dialog. The cluster terminates and your kernel returns to the default compute pool.
Shared cluster (many notebooks on one cluster)
With a shared template, a single underlying Kubernetes cluster can host the drivers and executors of many notebooks at the same time. The cluster is the shared resource pool. Each attached notebook still gets its own kernel pod and its own Spark driver — only the underlying cluster is shared. Variables, imports, and cell state never leak between notebooks; as well, identity is per-notebook (the user who attached this notebook is the user this notebook's kernel runs as). Sharing applies to the compute pool, not to the kernels running on it.
- Pick the shared template from the template dropdown.
- If a shared instance is already running, the toolbar auto-selects it. + Create New Instance is greyed out so you don't accidentally spin up a duplicate.
- Click Attach. The status indicator shows Attaching → Connected as your kernel moves onto the cluster.
- Run cells. Other notebooks may also be attached to this cluster — they don't see your code, your variables, or your output. They share the underlying compute, not the kernel.
- When you're done, click Detach. Confirm the dialog. Your notebook disconnects from the cluster — the cluster keeps running for any other notebooks still attached, and for anyone who attaches in the future.
Good to know — kernel state is lost on Attach, Stop, and Detach
Each of these actions restarts your kernel pod (it has to move between compute pools). Variables, imports, and any in-memory data are cleared. Plan to re-run your setup cells after attaching, stopping, or detaching.
Idle timeout for shared clusters
Every runtime template specifies an idle timeout. A shared cluster auto-stops once both of these conditions are true for the timeout period:
- No notebooks are attached to it.
- There is no Spark activity on the cluster.
If you only need to step away for a moment, leave your notebook attached — the cluster will stay up. If you're walking away for the day, click Detach so the cluster can shut down once no one else is using it.
Persistence across browser refresh
The toolbar remembers which template and instance you last used. When you reopen a notebook, the template is auto-selected and the toolbar polls for the instance:
- If the instance is still Running, the toolbar offers Attach so you can pick up where you left off.
- If the instance was provisioning when you closed the tab, the toolbar resumes polling.
- If the instance is gone (stopped externally, idle-timed-out), the toolbar falls back to + Create New Instance.
Seeing a shared cluster from another notebook
If Alice attaches Notebook A to a shared cluster, Bob will see the same cluster in his instance dropdown when he opens Notebook B and picks the same template. He can attach to it directly. There is no real-time sync — Bob will not see Alice's actions live; he has to reopen the dropdown to refresh the list. The status indicator does poll independently, so if the cluster is stopped externally, both Alice's and Bob's toolbars notice and update on their next poll.
Templates that don't appear in the dropdown
If a runtime template you expect to see is not in your dropdown, the most likely reason is that the template has not been shared with you. Templates the user has access to are shown; everything else is filtered out. Ask your admin if you need access to a template that isn't visible.
If no templates appear at all, none have been published yet — again, an admin task. The default kernel still works in the meantime.
Errors and recovery
If something goes wrong, the status indicator turns red and a tooltip shows the error message. The most common errors and what to do about them:
| Error | What it means | What to do |
|---|---|---|
| Failed to load templates / instances | The toolbar couldn't reach the platform's runtime service. | Refresh the page. If the error persists, contact your admin. |
| Cluster creation failed | The runtime service couldn't provision the cluster (quota, image pull, or networking). | Read the error message for specifics. Try a different template; escalate to your admin if the same template keeps failing. |
| Cluster creation timed out | Provisioning didn't finish within 15 minutes. | Click Start again. If it still times out, try a different template — the underlying compute pool may be at capacity. |
| Runtime stopped (idle timeout or external action) | The cluster you were attached to has been stopped. | Click Start to spin up a new instance, or Attach to a different running instance. |
| Kernel pod is on the wrong node pool | The kernel was rescheduled but did not land on the runtime's compute pool. | Click Attach again. If it keeps failing, restart the kernel from the kernel menu and re-attach. |
| Kernel is dead — bootstrap likely failed | The kernel pod started but the Syntasa kernel bootstrap script crashed. | Open the kernel logs viewer (see Kernel logs Viewer) and look for the failure — a bad init script or missing dependency is the usual cause. |
| Status poll failed | Background polling can't reach the runtime service. | Refresh the page if the error sticks; usually transient and the toolbar retries automatically. |
Related sections
- Spark Session Information. Once attached, a styled HTML block in your notebook output summarizes what the new Spark session is running on.
- Notebook Process (Jobs). Notebook Process can specify a runtime template at submission time so scheduled runs use the same compute as your interactive sessions.
- Init Scripts & Dependencies. A runtime template can carry its own init script and dependency configuration that run at Spark session creation.
- Kernel Logs Viewer. If a kernel fails to start after Attach, the Bootstrap tab of the logs viewer is the fastest way to find out why.
- Spark UI Access. Once your Spark session is running, the toolbar's Spark UI button opens the Spark application web UI.