How do I install "offline" dependencies at runtime?

Completed

Comments

1 comment

  • Avatar
    Mike Z

    This can be done by doing a combination of storing your package files in your data store (i.e. S3, GCS, Blob) and adding a parameter to the Spark runtime.

    1. Download the package files (i.e. .whl)
    2. Upload the files in the /syn-cluster-config/deps/python/ folder in your bucket
    3. Add two new Config Settings/Values to your runtime (syntasa.python.enable.dependencies=true and 

      syntasa.python.dependencies.names=<package1,package2....>

    4. When you start a cluster using the runtime with the new config settings the specified packages will be installed

    See sample screenshot below of the two settings.

     

    0
    Comment actions Permalink

Please sign in to leave a comment.