Lib Utilities

lib — install packages and JARs

Python — installPyPI / installCondaPackage

 

Method

Purpose

installPyPI(packages, acrossAllNodes=True)

pip install. Auto-downloads s3:///gs:///https:// URLs, plus .zip / .whl / .tar.gz filenames from the cluster's <bucket>/<config-folder>/deps/python/ folder. Multiple packages space-delimited.

installCondaPackage(packages, acrossAllNodes=True)

conda install (channel conda-forge). Multiple packages space-delimited.

Set acrossAllNodes=False to install on the kernel only (skip distribution to Spark worker nodes).

 
# 1. Regular PyPI package synutils.lib.installPyPI("requests") 

# 2. Multiple packages synutils.lib.installPyPI("requests pandas>=2.0 numpy") 

# 3. Local zip from the cluster's deps/python folder 

# (resolves to <bucket>/<config-folder>/deps/python/simple_module.zip) 
synutils.lib.installPyPI("simple_module.zip") 

# 4. Full cloud path (s3://, gs://, or https://) 
synutils.lib.installPyPI("s3://syntasa-k3s-dev/pradeepm/python_modules/simple_module.zip")

# 5. Mix everything in one call 
synutils.lib.installPyPI("requests simple_module.zip s3://syntasa-k3s-dev/pradeepm/python_modules/other.whl") 

# 6. Kernel-only install (don't distribute to Spark workers) 
synutils.lib.installPyPI("requests", acrossAllNodes=False) 

# 7. conda-forge package 
synutils.lib.installCondaPackage("scipy")

Scala — installJars

Install JARs into the running Scala kernel.

Accepts three source styles, mixed freely in a single space-delimited call:

Source

Example

Maven coordinates

"org.joda:joda-money:1.0.4"

Filename in the cluster's deps folder (<bucket>/<config-folder>/deps/jars/)

"greeterWithDollar.jar"

Full cloud-storage path

"gs://syn-400-development-kub/pradeepm/greeter.jar"

 
 
// Single source synutils.lib.installJars("org.joda:joda-money:1.0.4")
 synutils.lib.installJars("greeterWithDollar.jar") 
 synutils.lib.installJars("gs://syn-400-development-kub/pradeepm/greeter.jar") 
 
 // Multiple, space-delimited 
 synutils.lib.installJars("org.joda:joda-money:1.0.4 greeterWithDollar.jar gs://syn-400-development-kub/pradeepm/greeter.jar")