Using Zeppelin Notebooks in Syntasa

Article Summary

  1. Create a Zeppelin Runtime (if one does not already exist)
  2. Launch Zeppelin
  3. Create a Zeppelin Notebook (Default & Big Query)

Overview

Establishing a Runtime that is compatible with running Zeppelin Notebooks is a one-time activity. Please verify a Zeppelin-compatible Runtime does not already exist before creating and performing the following steps.

Use of Zeppelin requires a cluster to be up and running, there are no cluster size requirements to make use of Zeppelin, so best practice is to use a very minimal amount of resources. We typically use a two-worker cluster using n1-highmem-2 nodes on GCP and use the equivalent on AWS, costing roughly $0.15 per node per hour. In this configuration, the cost of operation would be in the range of $0.40-$0.50 per hour.

Also, we highly recommend setting a max uptime for the cluster, so if the cluster does not get shut down by the user, it will automatically shut down after the specified max uptime has been reached. 

Creating a Zeppelin Runtime 

 

Please follow the below steps for creating a Zeppelin-compatible Runtime. If the Runtimes already exist, please move to the next section.

  1. Create a new Runtime by copying the default Spark cluster.
  2. Turn OFF the ‘Terminate on Completion’ button and turn ON the ‘Zeppelin’ button.
  3. Update the cluster base name to ‘zeppelin – cluster’.
  4. Update the master and worker instances type to a tiny instance type (Recommended: n1-highmem-2).
  5. Update HDD to 100 GB.
  6. Update the Worker Instance count to 2 (minimum workers require for the cluster in GCP)
  7. Update the Idle Time per your requirement (suggested 5hrs - 18000 sec)
  8. Update spark configurations as below, remove all other configs.
    • Executor memory: 2G
    • Driver memory: 2G
    • Cores: 1
  9. Click on save to create the Runtime.
  10. Following is the sample Zeppelin Runtime screenshot:

 mceclip0.png

 Launch a Zeppelin Notebook

  1. Turn on interactive mode 
  2. Create or modify runtime with the Zeppelin runtime.Screen_Shot_2020-02-04_at_1.56.50_PM.png
  3. Select the Zeppelin link after the cluster has been created (could take as long as a few minutes)Screen_Shot_2020-02-04_at_2.02.33_PM.png
  1. Navigate to the Zeppelin home page using the link from the above step.mceclip3.png
  2. steps Update GCP Project Details in Interpreter Settings.  
    • Click on the ‘anonymous’ user on the rightmost corner of the page and select ‘Interpreter’.
    • Search for BigQuery interpreter
    • Edit the BigQuery interpreter properties and add the GCP project id. Save the changes.

 mceclip4.png

mceclip0.png

Create a Zeppelin Notebook

  1.  Select the ‘Notebook’ option from the main menu
  2. Create a New notebook or select an existing notebook from the list.

 mceclip5.png

  1. Create a New Notebook:
    • Click on ‘Create new note’ to start a new notebook.
    • Name your notebook and select Default (Spark) or  BigQuery as the default interpreter. Hit on Create button.

mceclip6.png 

  • The following screen will appear, here you can start writing queries and execute them.

 mceclip7.png

  • Add '%pyspark' if you're using spark or  ‘#standardSQL’ if you chose Big Query as your default editor as a tag before your query. Hit on the run button or shift+enter to execute the query.

 mceclip8.png

  • Add new paragraphs to run more queries.
  • Use a markdown interpreter to write comments. Add ‘%md’ as the first line to use the markdown interpreter. Below is the sample screenshot.

 mceclip9.png