Syntasa 9.0.1

This update introduces necessary bug fixes and enhancements to Spark-on-Kubernetes (K8s) workloads, particularly around dependency management, initialization scripts, and runtime efficiency for both batch jobs and interactive notebooks.

Spark-on-Kubernetes Enhancements

Prebaked Python Environments & Faster Startup

To reduce job startup times, Spark-on-K8s batch jobs now utilize pre-installed Python environments for versions 3.7, 3.9, 3.10, and 3.11. This eliminates the 2–3 minute delay previously required to build Conda environments at runtime. The correct Python version is now activated instantly based on the job configuration.

Unified Dependency & Init Script Management

Initialization scripts and dependency configurations (pip packages, JARs, Maven coordinates) are now fully supported on Kubernetes for both driver and executor pods:

  • Global & Runtime Init Scripts - Scripts configured at the platform or runtime level now execute on K8s, with output streamed directly to the Syntasa UI log viewer.
  • Executor Support - K8s executors now run global, notebook, and runtime init scripts during bootstrap.
  • Dependency Configs - New Spark properties (`syntasa.python.*` and `syntasa.jar.*`) allow for automated downloading of JARs to `$SPARK_HOME/jars/` and pip installation of Python packages across the cluster.
  • Resilience - Init script failures are now non-fatal; if a script exits with an error, the job will continue to launch with a warning logged.

Improved Error Reporting

  • JAR Installation - Fixed an issue where JAR files specified in the runtime configuration were not being correctly installed on K8s clusters.
  • Clearer Logs - Improved the clarity of error messages when an invalid class is imported from a JAR. Instead of a generic "Compilation Failed" message, the logs now provide specific details (e.g., `ClassNotFoundException`) in both short and detailed views.

Notebook & Interactive Improvements

Runtime Reuse for Notebooks

Notebook process jobs and notebook cards can now reuse existing, running Spark-on-K8s runtimes. Previously, these jobs would always trigger the creation of a new cluster, leading to higher costs and longer wait times.

  • The "Select an instance" dropdown in the Notebook Runtime Settings now displays all running instances from the same template.
  • Fixed logic for "Terminate on Complete = False" to ensure idle clusters scaling down are correctly intercepted and reused by new jobs.

Scala Kernel Parity

The Scala notebook kernel now shares the same robust initialization logic as the Python kernel. It supports runtime init scripts, dependency configurations, and automatic classpath inclusion for JARs downloaded via scripts.

Ad-hoc Package Installation

`synutils.installPyPI` and `synutils.installCondaPackage` have been optimized for K8s. They no longer trigger a full `conda_pack` (saving 30–90 seconds). Instead, ad-hoc packages are tracked and installed on executor pods during the bootstrap phase.

Bug Fixes & Stability

  • Custom Script Formatting - Resolved an issue where custom runtime scripts lost indentation and line endings when saved to S3, which previously caused bash syntax errors.
  • Instance Visibility - Fixed a UI bug in Notebook cards where newly created runtime instances were not visible in the dropdown until they reached a "Running" state, preventing users from starting them.
  • Conda Override - Addressed a failure in AWS environments where setting `syntasa.override.conda.env = true` caused `boto3` related errors.