How do I find out what Python modules are installed on the Spark cluster connected to my Notebook?

Pre version 7.0

I've tried code that works in the Spark Processor (see below) in my notebook but keep getting errors. What is the correct code to validate dependencies and versions that are installed on the Spark cluster connected to my notebook?

Code Run (works in Spark Processor, not Notebook)

%%spark
import os
x = os.system("pip freeze")
print(x)

Example Error (the installed package displayed changes each time run)

An error was encountered:
Unrecognized token 'cachetools': was expecting ('true', 'false' or 'null')
 at [Source: cachetools==5.0.0; line: 1, column: 11]

Mahesh Shenoy July 29, 2022 09:06

Mike Z was this question addressed offline? If you had identified it, would be great if you could add it over here so that anyone else can refer later.

Sarath Botlagunta / Shahdy Ali Hassan

Mahesh Shenoy November 17, 2022 09:12

Marking completed as Mike Z got the info and added the details.

Shahdy Ali Hassan

How do I find out what Python modules are installed on the Spark cluster connected to my Notebook?

Comments

Didn't find what you were looking for?