How do I find out what Python modules are installed on my Spark cluster?Completed
This code provides one way to determine what Python modules are installed on the running Spark cluster. Place this code in a Spark processor and run a job. When the job completes, check the logs for a list of Python modules
x = os.system("pip freeze")
Please sign in to leave a comment.