Notebook Utilities Reference

Syntasa Notebook Utilities Reference — Python & Scala

synutils is a pre-initialized utility object available in every Syntasa Notebook. It provides access to credential management, cloud storage, infrastructure configuration, Spark initialization, package installation, and more.

# synutils is already available — no import needed
synutils.credentials.list()
synutils.fs.upload("/tmp/data.csv", "my-bucket/data.csv")
synutils.lib.installPyPI("pandas")

Available Modules

Credential Store

Securely store, retrieve, update, share, and manage credentials (API keys, passwords, tokens) using HashiCorp Vault. Credential values are automatically redacted in notebook output to prevent accidental exposure.

create Both

Create a new credential store.

synutils.credentials.create("my_api_keys", {
    "api_key": "abc123",
    "api_secret": "xyz789"
})

read Both

Read credentials. Returns all keys as a redacted dictionary, or a single redacted value.

# Read all keys — returns SecretDict
creds = synutils.credentials.read("my_api_keys")
# {'api_key': '[REDACTED]', 'api_secret': '[REDACTED]'}

# Read a specific key — returns SecretString
api_key = synutils.credentials.read("my_api_keys", key="api_key")
# [REDACTED]

Using Credentials with SDKs

creds = synutils.credentials.read("aws_creds")
raw = creds.unseal()  # Returns plain dict with real values

client = boto3.client('s3',
    aws_access_key_id=raw['access_key'],
    aws_secret_access_key=raw['secret_key']
)

update Both

# Update a single key
synutils.credentials.update("my_api_keys", key="api_key", value="new_key_123")

# Update multiple keys
synutils.credentials.update("my_api_keys", updated_kv={
    "api_key": "new_key_123",
    "api_secret": "new_secret_456"
})

delete Both

synutils.credentials.delete("my_api_keys")

list Both

Lists all credential stores for the current user. Python prints a formatted PrettyTable; Scala prints names to stdout.

synutils.credentials.list()
# +----------------+
# |  Credentials   |
# +----------------+
# | my_api_keys    |
# | db_passwords   |
# +----------------+

share Both

synutils.credentials.share(
    username="colleague@company.com",
    permissions=["read"],
    credential_name="my_api_keys"
)

revoke Both

Revoke a user's access. Same parameters as share.

File System

Cloud storage operations for uploading, downloading, listing, and managing files. The implementation is automatically selected based on your cloud provider (AWS S3, Google Cloud Storage, or Azure Blob Storage).

upload Both

synutils.fs.upload("/tmp/report.csv", "my-bucket/reports/report.csv")

upload_folder / uploadFolder Both

synutils.fs.upload_folder("/tmp/output/", "my-bucket/output/")  # Python
synutils.fs.uploadFolder("/tmp/output/", "my-bucket/output/")   // Scala

download Both

synutils.fs.download("my-bucket/data/input.csv", "/tmp/input.csv")

download_folder / downloadFolder Both

list Both

files = synutils.fs.list("my-bucket/data/")

list_file_paths Python

List full file paths in a directory. Available on S3 and Azure.

is_exists / isExists Both

if synutils.fs.is_exists("my-bucket/data/input.csv"):
    print("File found!")

delete Both

synutils.fs.delete("my-bucket/old-data/temp.csv")

move Python

Move or copy files within cloud storage.

synutils.fs.move("my-bucket/temp/", "my-bucket/final/")

# With exclusions
synutils.fs.move("my-bucket/src/", "my-bucket/dest/",
    exclude_folders=["logs", "temp"])

stream Python

Stream file content from cloud storage. Returns a file-like object. Available on S3 and GCS.

upload_stream Python

Upload from a file-like object to cloud storage.

read Scala

Read file contents as a byte array.

val content: Array[Byte] = synutils.fs.read("my-bucket/config.json")
val text = new String(content, "UTF-8")

Scala-Only Properties

synutils.fs.PREFIX           // "s3://", "gs://", etc.
synutils.fs.getBaseDir()     // User's base workspace directory
synutils.fs.getLocalTempDir() // Local temporary directory

Infrastructure

Access platform infrastructure configuration: cloud provider type, region, storage, metastore, network, and security settings.

synutils.infrastructure.get_provider_type_from_metadata()
# Returns: "GOOGLE", "AMAZON", or "AZURE"

synutils.infrastructure.get_config_from_metadata()     # Region, project ID, etc.
synutils.infrastructure.get_storage_from_metadata()    # Bucket names, paths
synutils.infrastructure.get_metastore_from_metadata()  # Hive, Glue, BigQuery settings
synutils.infrastructure.get_security_from_metadata()   # Cloud credentials, SSH
synutils.infrastructure.get_network_from_metadata()    # Network config
synutils.infrastructure.get_global_init_script_from_metadata() # Init script or None

Library Management Python

Install Python packages at runtime. Packages are installed locally and automatically distributed to Spark executor pods on Kubernetes.

installPyPI

synutils.lib.installPyPI("pandas")
synutils.lib.installPyPI("pandas numpy scikit-learn")  # Space-separated
synutils.lib.installPyPI("requests>=2.28")              # Version constraint

installCondaPackage

synutils.lib.installCondaPackage("scipy")
synutils.lib.installCondaPackage("scipy numpy")  # Space-separated

Installs from the conda-forge channel.

Kernel Utilities Python

restart_kernel

synutils.kernel.restart_kernel()                        # Simple restart
synutils.kernel.restart_kernel(execute_cell_below=True) # Restart + run cells below

Spark Initialization

Spark is automatically initialized when your notebook starts. The spark variable (SparkSession) is available in your notebook namespace once initialization completes.

Python — Manual Control

synutils.start_spark_initialization()   # Re-trigger initialization
synutils.cancel_spark_initialization()  # Cancel initialization

Python — Advanced Configuration

from syn_utils.syn_notebook.utils.spark.spark_helper_utils import SparkHelperUtils

# Auto-detect resource manager
spark = SparkHelperUtils.initialize(driver_memory="4g", executor_memory="8g")

# Kubernetes-specific
spark = SparkHelperUtils.initialize_with_kubernetes(driver_memory="4g", executor_memory="8g")

# YARN-specific (requires cluster master IP)
spark = SparkHelperUtils.initialize_with_yarn(syntasa_cluster_master_ip="10.0.0.1")

# Local mode
spark = SparkHelperUtils.initialize_local_spark(driver_memory="4g")

# Ray-based Spark
spark = SparkHelperUtils.initialize_ray_spark(
    driver_memory="4g", driver_cores="2",
    executor_memory="8g", executor_cores="4")

Scala — SparkInitializer

import com.syntasa.kernel.utils.spark.SparkInitializer

SparkInitializer(
    name = "MyNotebook",
    resourceManager = "KUBERNETES",  // or "YARN", "LOCAL"
    driverMemory = Some("4g"),
    executorMemory = Some("8g"),
    executorCores = Some("4"),
    userConfiguration = Map("spark.sql.shuffle.partitions" -> "200")
)

// `spark` is now available globally
spark.sql("SELECT * FROM my_table")

SparkInitializer Parameters (Scala)

Ray Cluster Management Python

Create and manage Ray clusters for distributed computing.

create_cluster

cluster = synutils.ray.create_cluster("my-ray-template")

ray_init

Initialize connection to a Ray cluster.

setup_spark

spark = synutils.ray.setup_spark()
spark = synutils.ray.setup_spark(debug=True, additional_spark_config={"spark.executor.memory": "8g"})

Returns: SparkSession

delete

Delete a Ray cluster and its resources.

GeoMesa (Geospatial) Both

Register GeoMesa/Sedona geospatial UDFs in your Spark session for spatial queries. Functions are registered with the sedona_ prefix.

synutils.geo_lib.register(spark)
df = spark.sql("SELECT sedona_ST_Point(lon, lat) as geometry FROM locations")

Run Notebook Python

Execute another notebook in the current kernel context. The notebook is downloaded from cloud storage and run locally. Variables and functions defined in the target notebook become available in the current session.

synutils.run_notebook("my_folder/shared_functions.ipynb")

Authentication Both

Access and manage authentication tokens. Primarily used internally, but available for making authenticated API calls.

# Token access
token = synutils.auth.get_access_token()
refresh = synutils.auth.get_refresh_token()
auth_type = synutils.auth.get_auth_type()

# Token validation & refresh
valid_token = synutils.auth_client.validate_token_from_metadata()
result = synutils.auth_client.validate_token(token)

# Login
response = synutils.auth_client.login("user@company.com", "password", "LOCAL")
new_tokens = synutils.auth_client.refresh_token("LOCAL", refresh_token)

# User info
user_info = synutils.auth_client.get_user_info_by_name("user@company.com")

Configuration Python

synutils.configuration.get("key_name")
synutils.configuration.set("key_name", "value")

provider = synutils.configuration.get_cloud_provider_type()
# Returns: "GOOGLE", "AMAZON", or "AZURE"

# Static method — get environment variable
from syn_utils.syn_auth.config import Configuration
env_val = Configuration.get_env("MY_ENV_VAR", default="fallback")

Quick Reference

# --- Credential Store ---
synutils.credentials.create("name", {"key": "val"})
synutils.credentials.read("name")                          # → SecretDict
synutils.credentials.read("name", key="key")               # → SecretString
synutils.credentials.update("name", key="k", value="v")
synutils.credentials.update("name", updated_kv={"k": "v"})
synutils.credentials.delete("name")
synutils.credentials.list()
synutils.credentials.share("user", ["read"], "name")
synutils.credentials.revoke("user", ["read"], "name")

# --- File System ---
synutils.fs.upload("/local/file", "remote/file")
synutils.fs.download("remote/file", "/local/file")
synutils.fs.upload_folder("/local/dir/", "remote/dir/")
synutils.fs.download_folder("remote/dir/", "/local/dir/")
synutils.fs.list("remote/path/")
synutils.fs.list_file_paths("remote/path/")
synutils.fs.is_exists("remote/file")
synutils.fs.delete("remote/file")
synutils.fs.move("src/", "dest/")
synutils.fs.stream("remote/file")
synutils.fs.upload_stream(file_obj, "remote/file")

# --- Infrastructure ---
synutils.infrastructure.get_provider_type_from_metadata()
synutils.infrastructure.get_config_from_metadata()
synutils.infrastructure.get_storage_from_metadata()
synutils.infrastructure.get_security_from_metadata()
synutils.infrastructure.get_metastore_from_metadata()
synutils.infrastructure.get_network_from_metadata()
synutils.infrastructure.get_global_init_script_from_metadata()

# --- Library Management ---
synutils.lib.installPyPI("pandas numpy")
synutils.lib.installCondaPackage("scipy")

# --- Kernel ---
synutils.kernel.restart_kernel()
synutils.kernel.restart_kernel(execute_cell_below=True)

# --- Spark ---
synutils.start_spark_initialization()
synutils.cancel_spark_initialization()

# --- Other ---
synutils.run_notebook("path/to/notebook.ipynb")
synutils.geo_lib.register(spark)
synutils.ray.create_cluster("template")
synutils.ray.setup_spark()

Workflow Example

# 1. Install a package you need
synutils.lib.installPyPI("snowflake-connector-python")

# 2. Get credentials for Snowflake
sf_creds = synutils.credentials.read("snowflake_prod")
raw = sf_creds.unseal()

# 3. Connect using credentials
import snowflake.connector
conn = snowflake.connector.connect(
    user=raw["user"],
    password=raw["password"],
    account=raw["account"],
    warehouse=raw["warehouse"],
    database=raw["database"]
)

# 4. Query and load into Spark
df = spark.read.format("snowflake") \
    .options(**raw) \
    .option("query", "SELECT * FROM events") \
    .load()

# 5. Save results and upload
result = df.toPandas()
result.to_csv("/tmp/results.csv", index=False)
synutils.fs.upload("/tmp/results.csv", "my-bucket/output/results.csv")