This article explores the Deploy feature within the Syntasa platform, a tool for building and deploying behavioral data pipelines. In this article, we will cover:
- Understanding Development Workflow
- Understanding Production Workflow
- Role of the 'Deploy' Feature
- Relevance of the 'Deploy' feature
- When to use the 'Deploy' feature?
- How to Use the 'Deploy' feature?
- Understanding the 'Deploy' job
Development & Production Workflow
Syntasa provides distinct workflows for development and production environments, ensuring a clear separation between the stages of creating, testing, and deploying applications. Before delving into the deployment process, let's first understand both the development and production workflows.
Development Workflow
The development workflow is designed for creating, experimenting, and refining your application without impacting the live environment. In this phase, users can:
- Create and test workflows and models.
- Modify configurations and logic freely.
- Conduct extensive testing and validation using sample data.
The development environment allows flexibility in making changes and iterating over the application design. It includes features like version control and the ability to take snapshots, which capture the state of the app at different points in time for easy rollback if needed.
Production Workflow
Once the application has been thoroughly tested and is ready for deployment, it moves to the production workflow. In this environment:
- The workflow is locked to prevent unauthorized changes.
- Only job creation and scheduling are allowed.
- Jobs can be scheduled for recurring execution, providing automation for regular data processing tasks.
The production workflow ensures stability and reliability for live data processing and model execution. It also supports job monitoring and alerting to keep track of the application's performance and any potential issues.
By maintaining separate environments for development and production, Syntasa ensures that development activities do not interfere with the production environment, thereby providing a robust framework for both experimentation and reliable production operations.
Role of 'Deploy' Feature
Deploy is a functionality within Syntasa that plays a crucial role in transitioning your data models from the development phase to a production environment. It essentially performs two critical actions:
- Copies Apps: It transfers your applications developed in the Syntasa development workflow over to the production workflow. These applications likely involve data ingestion, transformation, and model creation.
- Updates Table Schema: If any modifications were made to table structures during development, Deploy handles updating the table schema within the production environment to reflect these changes.
Relevance of the 'Deploy' Feature
Deploy allows you to seamlessly move your data pipelines from development to production, where they can start processing real-time data and generate valuable insights. This facilitates the practical application of the models you've built within Syntasa.
When to use the 'Deploy' feature?
The Deploy feature in Syntasa should be used when you want to transition your data pipelines from the development environment to the production environment. This is typically done after you've completed the following steps in the development workflow:
- Configured data sources: You've established connections to all the data sources your pipelines will need to ingest data from.
- Built data transformations: You've created the logic to transform your raw data into a usable format for analysis. This may involve filtering, aggregating, or enriching the data.
- Created and trained models: You've built and trained the models you'll be using to analyze the data. These models could be for tasks like behavioral analytics, anomaly detection, or customer segmentation.
- Thoroughly tested: You've rigorously tested your data pipelines and models within the development environment to identify and fix any errors or inefficiencies.
Once you're confident that your data pipelines and models are functioning as expected in the development environment, then it's time to leverage the Deploy feature. Deploy will push your pipelines and any associated schema changes from development to production, enabling them to start processing real-time data and generating valuable insights.
How to use the 'Deploy' feature?
Once you're confident that your data pipelines and models are functioning as expected in the development environment, then you can click on the 'Deploy' button shown on the top right side as highlighted below:
When you click the 'Deploy' button, a screen will display all the datasets ready for deployment to production. For the first deployment, no action is needed—just click 'Deploy' to start the process. After this initial deployment, you can edit the copy mode for each dataset. Let's review each column on this screen:
- Name: Represents the name of the dataset.
- Object type: Indicates the type of object the dataset represents. Common object types include Table, View, and Lookup.
- Change: It indicates whether this dataset has been modified or not.
-
Copy Mode: Determines the action to be taken on the dataset during deployment. You can modify the Copy Mode by clicking the 'Edit' icon. There are different copy mode options depending on the type of template you have selected for an app. The available options are:
- None: No action is taken with table definitions in the backend. This is used when no changes are made to dataset columns.
- Drop Copy: Drops the existing model and copies the new one.
- Alter: Alters the table definition with changes from the development table. This is used when new columns are added or deleted, or when the data types of existing columns are changed.
- Alter Copy: Modify the dataset and then copy it.
- Drop Create: Remove the existing dataset and create a new, empty one.
- Drop Create Copy: Delete the dataset, create a new schema, and copy data from the source.
- Create Copy: Create a new schema and copy the data.
- Drop Copy: Remove the existing model and copy a new one.
- Copy: Copy the existing model.
Understanding the 'Deploy' Job
The Deploy feature in Syntasa performs two main functions:
- Copy the Development Workflow to Production: This ensures that the latest version of the workflow is moved to the production environment.
- Alter Table Schema: Where applicable, it modifies the table schemas to accommodate any changes without requiring a complete reconstruction of the tables.
When you click 'Deploy', from the above screenshot, a job named 'Deploy' is created and begins running. This job handles the schema alterations (step 2), ensuring that any new columns are integrated seamlessly. The application detects these schema changes and adjusts the dataset copy mode accordingly.
You can find the 'Deploy' job status execution by navigating to Operations>>Jobs under Development workflow. In the next article, we will cover the snapshot feature where a snapshot is automatically taken when an app is deployed to production and we can roll back the same using the Load feature.