Interactive Mode

The Interactive Mode toggle (Interactive) on the Horizontal bar of the App Canvas screen allows users to turn on/off Interactive Mode. This enables the ability to start and leave a cluster up while building or editing an app (only available in Development workflows).

Turning on the Interactive Mode

Once the interactive mode is turned on, a right-side panel opens with the 'Configuration' section. The 'Run' option appears on the top bar and action buttons related to non-interactive mode like Job, Replay, Deploy, Snapshot, Load etc disappear. 

This right-side panel has the following 4 sections for different purposes:

• Notes
• Activity
• Configurations
• Process Configurations

The last two sections, namely Configuration and Process Configurations, are accessible only in Interactive Mode. Consequently, these sections are visible only when Interactive Mode is activated. The 'Activity' section, on the other hand, is consistently accessible, but it displays activities related to Interactive Mode when it's enabled and non-interactive mode activities when it's disabled.

We will go through each section in the following order to understand the features easily: Configurations, Process Configurations, Activity.

Configurations

Once the Interactive Mode toggle turns to ON, by default system opens the configuration section in the right pane.

In the configuration Section, a new Runtime can be added by clicking the '+ Add Runtime' link that will allow you to give a Name to the Runtime, and also select the runtime template depending on your environment.

For example: If you are using GCP, you can see all the runtime templates you have added for the runtime types GCP DataProc Cluster and Kubernates Container. If you are on the AWS environment, you can see all the runtime templates you have added for the runtime types AWS EMR Cluster and Kubernetes Container.  

If you want to learn how to create a runtime template, you can check this article.

Let's start with different types of runtime that we can add and what features they offer. We will come back to the configuration settings showing on the top of the screen later. 

Kubernetes Container:

When you click 'Add Runtime', you'll be prompted to input a Name and choose a Runtime template. Under the 'Runtime template' dropdown options, you can find all the runtime templates of runtime type 'Kubernetes Container' along with other runtime types. You can select your created runtime template and save it. Once the runtime is added, the runtime will automatically be created and start running. You can stop or restart the runtime as per your requirement.

You can add multiple runtimes of runtime type 'Kubernetes Container' irrespective of the image type. Just so you know, the current limit of adding runtime under interactive mode is currently set to 6 irrespective of the runtime type.

Kubernetes container node groups scale down to 0 when there is no container interactive job being executed. This is similar to idle timeout on 'GCP DataProc Cluster' and it doesn't incur additional cost.

GCP DataProc Cluster:

When you select 'Add Runtime', you'll be prompted to input a Name and choose a Runtime template. If you choose a runtime template of 'GCP DataProc Cluster', the default settings for that template will appear in disabled mode. You can access these settings by navigating to Resources >> Runtime.



Under Default Settings, the 'Google DataProc Cluster' runtime provides two extra functionalities to halt the cluster under specific conditions. These capabilities offer significant cost-saving benefits by stopping the cluster when it's not in use, thereby optimizing resource utilization. Here are these two functionalities:

Runtime Max Uptime & Units: This parameter indicates the maximum runtime duration. Once this limit is reached, regardless of the job's status, the runtime will cease. For instance, in the screenshot above, the runtime will halt after running for 1 hour, with its status changing to 'Stopped'.

Idle Time Deletion: This parameter signifies the duration of runtime inactivity before it automatically stops. When no jobs are running, and the runtime remains idle for the specified time, its status will transition to 'Stopped'. For example, in the provided screenshot, the runtime will stop after being idle for 600 seconds.

While adding the runtime, you have the option to overwrite the default settings of the runtime template by toggling 'Override Template Defaults'. You have the option to overwrite the default settings of the runtime template even after adding it. When the runtime is not running, you can click on the 'Edit' icon to overwrite the settings.

If you want to change the settings permanently, you can change that by selecting your runtime under Resources >> Runtime. Alternatively, you can visit the runtime template configuration page directly by clicking the runtime link shown after adding the runtime.

 

Once the runtime is added, the runtime will automatically be created and start running. You can stop or restart the runtime as per your requirement. 

When the runtime is running, you can expand the runtime to check default settings and cluster running time. If you have enabled Zeppelin and Jupyter Notebook for the selected runtime template, you will find links to Zeppelin, Jupyter, and Jupyter Lab on expanding the runtime during 'Running' status. 

Note: Unlike Kubernetes Container, you can only add one runtime using a runtime template with the runtime type 'GCP DataProc Cluster'. Attempting to add a second similar runtime will result in an error, as depicted below.

 

Configuration Fields

Under the 'Configurations' screen, located at the top, you have the option to customize settings for executing processes. You can adjust the following fields for all processes:

• Date Range: Specify the execution date range for processing data.
• Test File Limit: Set the limit for test files. The minimum limit is currently set to 1, while the maximum limit is set to 2147483647.
• Process Mode: Choose from four options for process mode: Drop & Replace, Replace Date Range, Add New & Replace Modified, Add New only, etc.

Once you save these details, they will be applied to any job run in interactive mode. These configuration settings persist even when an interactive mode is turned off, ensuring that the same settings are available when an interactive mode is reactivated.

Process Configuration

After adding Runtime under the 'Configuration' section, you can navigate to the 'Process Configurations' section to configure runtime for each process of the App. By default, each process will show the added runtime(s) compatible with the process. You can expand the process and select the runtime of your choice from the list. 

Activity

We can see all activities of the process under the 'Activity' section. After configuring the settings and starting the runtime, you can execute processes in three ways:

1. Run all processes together: To execute all the processes in the workflow simultaneously, you can click on 'Run' located at the top right corner of the screen (highlighted below).

Upon clicking 'Run', a new run will be initiated and displayed in the 'Activity' section. Once expanded, you can view all the configurations and details related to the run, including the status of each process. Additionally, you can expand each process to access details such as start time, end time, runtime type, runtime, data details, process mode, etc. For further insights, you can examine detailed logs for each process by clicking on the log icon associated with each one.

2. Run a single Process: To execute a single process, simply right-click on the desired process and select the 'Run' option. This option will be available for each process. Upon selecting 'Run', a new run will be generated under the 'Activity' tab, specifically containing only the chosen process.

3. Run selected processes: If you wish to execute only specific processes, right-click on each desired process and select the 'Run' option displayed on the last process to execute all the selected processes simultaneously. After selecting 'Run' on the last process, a new run will be created under the 'Activity' tab, including all the chosen processes.

 

Turning Off the Interactive Mode

Once you turn off the interactive mode, all running runtime will be shut down. The 'Run' option will disappear and action buttons related to a non-interactive mode like Job, Replay, Deploy, Snapshot, Load, etc will reappear. The 'Activity' section will start and stop activities related to the interactive mode and activities related to the non-interactive mode will be displayed.

When you turn on the interactive mode again, your all saved configurations related to run and process, runtime(s), activities, etc. will reappear.