Syntasa empowers data teams to build complex data pipelines with ease. However, keeping track of how data flows between various applications can become a challenge. This is where the App Dataflows screen comes in. Imagine a roadmap that visually depicts the intricate connections between your apps, allowing you to effortlessly trace the data's journey from ingestion to utilization.
In this article, we will cover the following topics:
- What are the benefits of dataflow?
- How to create a dataflow?
- Understanding Dataflow with an example
- Viewing App Details
- Viewing Connection Details
- How to update the Dataflow?
- How to delete the dataflow?
Benefits of App Dataflows
These dataflow visualizations offer a multitude of advantages, including:
- Troubleshooting Made Easy: Encountered an issue with your data pipeline? Visualizing the connections between apps can help you pinpoint bottlenecks or errors in data transmission, allowing for a quicker resolution.
- Optimizing Data Movement: By understanding data flow patterns, you can identify opportunities to streamline data movement within your pipelines, leading to increased efficiency.
Creating an App Dataflow
Here are the steps to create a new dataflow:
- Click on the 'Hamburger' menu and select 'Dataflow' under the 'Platform' section.
- Click on the '+' icon located on the right side, which will open the 'Create DataFlow' screen.
- Enter a unique name along with other necessary details.
- Click on 'Save'.
- Name: This is a unique name for the Dataflow.
- Description: Here, you can provide a description explaining the purpose of the dataflow.
- Apps: Select the apps to visualize the dataflow between the selected apps.
-
Home Screen (Toggle): Enabling this option will create a default homepage accessible by clicking the Syntasa logo. If no dataflow has this field toggled on, the below page will be displayed when clicking on the Syntasa Logo.
- Sharing Options: Select how you want to share the App dataflow.
Interacting with the App DataFlow Diagram
Once you have created a dataflow, you can select the dataflow on the grid to navigate to the App flow Diagram. The actions a user can take on the App Flow Diagram are to change the positioning of the apps on the diagram and view details on an app.
Understand App Dataflow With An Example
On this screen, users can visualize how all the selected apps are interconnected and adjust the positioning of the apps on the diagram. After rearranging the apps on the diagram, clicking the Save button will save the changes.
In the example below, there are two apps: 'Demo 1st App' and 'Demo 2nd App'. 'Demo 1st App' receives input from the connections 'UAT-S3' and 'QA_auto_AdobeApi'. 'Demo 2nd App' takes the output of 'Demo 1st App' as its input. Therefore, the data flows from 'Demo 1st App' to 'Demo 2nd App'.
Viewing Details of an App
Users can view the details of an app by clicking on the respective app, triggering a window to appear that provides app details under three tabs:
- Details: This tab displays the app name, template name, creation date, and app description. Users can also navigate to the application by clicking on its name.
- Jobs: Here, users can view the list of all production jobs associated with the selected app. For further details on job tracking, please refer to the Job Tracking article.
-
Dependencies: This tab presents a list of apps that the current app is dependent on. For instance, 'Demo 2nd App' relies on data from 'Demo 1st App'. Therefore, 'Demo 1st App' is shown listed under 'Dependencies'.
Viewing Connection Details
Similarly, you can also click on the connection and it will open a window showing connection details.
Editing an App Dataflow
You can update the dataflow details by clicking on the 'Edit' icon located in the last column of the record grid. This action will open the same screen as the 'Create Dataflow' screen, allowing you to update any information. Once you save the changes, the details and diagrams within the app dataflow will be updated accordingly.
Deleting an App Dataflow
To delete a dataflow, click the 'Delete' icon shown next to the 'Edit' icon on the dataflow list.