This code provides one way to determine what Python modules are installed on the running Spark cluster. Place this code in a Spark processor and run a job. When the job completes, check the logs for a list of Python modules
x = os.system("pip freeze")
Please sign in to leave a comment.