As described in Runtime Overview, runtimes are the compute resources used to execute processes and logic within jobs and notebooks defined within the Syntasa platform. The various tasks and experiments being done in apps and notebooks likely require different types and sizes of runtimes.
Creating a new runtime template entails several pieces of information, some only affecting the attributes of the runtime template seen within the Syntasa environment, and others relating to the cluster instance that is initiated within the cloud platform.
Runtime types
The first and most important decision when creating a new runtime template is the runtime type. The selection of runtime type affects the subsequent fields and settings available, which vary widely, therefore the details of each runtime type are detailed in dedicated articles.
Categorizing runtime types
While there are several runtime types available, these are categorized into two primary groups for easier understanding. Also, a third group is a slight variation of runtimes found in the two primary groups.
Container runtimes
Container runtimes are launched on the same Kubernetes cluster the Syntasa platform is using. They have limited to no settings to choose from because they are defined by the subsequently selected image. The image names, e.g. Syntasa Base Image or Syntasa Dash Image, help identify the intended use that the image is optimized, which is detailed in the article Kubernetes Container.
Cloud-native runtimes
Cloud-native runtimes, i.e. those with "cluster" in the name, utilize the appropriate cloud-native service (EMR in AWS; Dataproc in GCP; Synapse in Azure) to run jobs, execute code in notebooks, etc. These runtimes have all the various settings you would see in the cloud-native service. For example, the master and worker instance types, number of worker instances, max uptimes, etc.
Streaming runtimes
Streaming runtimes should only be used for streaming jobs, i.e. continuously running jobs unless manually stopped, as opposed to non-streaming runtimes that are used for batch jobs and notebooks that will be shut down once the job completes or inactivity or max timeout is reached.
Runtime types
The types of runtimes available depend on the cloud platform the Syntasa environment is installed; the settings available for the runtime template depend on the runtime type that is selected. The following runtime types are available for GCP and AWS:
Google Cloud Platform (GCP)
- GCP Dataproc Cluster
- Kubernetes Container
- GCP Dataproc Stream Cluster
- Kubernetes Stream Container
Amazon Web Services (AWS)
- AWS EMR Cluster
- Kubernetes Container
- AWS EMR Stream Cluster
Runtime attributes
The following fields define the runtime template object within the Syntasa application. The defined runtime templates are available for selection in jobs, notebooks, and the interactive mode of apps; the attributes affect how they are displayed in those areas.
- Name - Name as seen on the Runtime Template drop-down menus throughout the application. This name must be unique across all runtime templates.
- Copy Runtime - It may be needed to create a new runtime template that is slightly different from an existing one. Turning this on allows you to create a new runtime template from an existing one instead of starting from scratch.
- Description - Provide information explaining the purpose of the runtime template. This is especially useful if the runtime template is to be shared with other users.
- Tags - All objects throughout the Syntasa application support tags to help organize and search for related objects.
- Default Development/Production - Each runtime type + image type can have one runtime template denoted as the default for use in an app's development and/or production environment. These are used as the default runtime template assigned to the process/step of a job.
Each process (type) has a default runtime type + image type they should run on. Thus, per process/step of a job, the runtime template that gets assigned by default is the one that is of the appropriate runtime type + image type and has the relative environment checked.
Sharing and permissions
When creating a new app or resource, the sharing option selection will be available to make the component available as Private, Public, or Group. Regardless of the selection, a user with the role of System Admin will always have access to all apps and resources.
The sharing option selected at the time of creating the component can be changed by the owner or a system administrator. The owner of the app or resource is by default the user that is creating the component. This can be changed, if needed, by the owner or the system administrator, after the component is created.
When editing a component, the owner and sharing option is visible and can be changed by the owner or system administrator. For other users, when editing a component, the sharing option and owner settings are hidden.
- Private - Private can be set by the owner (or the system administrator) to limit access to the component only to the owner. System administrators can also access components set to private.
- Public - Public is the default setting and is the behavior of all components before Syntasa 5.2, i.e. before the sharing functionality was available, as all users have access to the component (except apps that are in a module, Synthesizer, Composer, Orchestrator, and the user's role is such that they do not have access to that module).
- Group - Group can be set by the owner (or the system administrator) to limit the access to the component to the user group(s) assigned to the component. As with private, the system administrator can view components with the sharing option set to group regardless if they are a member of the group.