How do I find out what Python modules are installed on my Spark cluster?

Completed

Follow

Mike Z

February 22, 2022 23:21

This code provides one way to determine what Python modules are installed on the running Spark cluster. Place this code in a Spark processor and run a job. When the job completes, check the logs for a list of Python modules 

import os
x = os.system("pip freeze")
print(x)

 

 

0

• 

  Mahesh Shenoy July 29, 2022 09:03

  Mike Z was this question addressed offline? If you had identified, would be great if you could add it over here so that anyone else can refer later.

  Sarath Botlagunta / Shahdy Ali Hassan

  0

  Comment actions Permalink
• 

  Mike Z November 16, 2022 23:59

  There are multiple ways this can be done. Please see the description section for one method.

  0

  Comment actions Permalink

Please sign in to leave a comment.