When working within a Syntasa application, many transformations and operations are performed using processes like Spark processor, From DB, transform etc . These steps often generate output saved into an Event Store. If this processed or transformed output needs to be sent back to a database for reporting, analytics, or integration with other systems, the TO DB process is used.
The TO DB process takes the data from an Event Store and writes it into a table in the target database.
Supported Databases
Syntasa supports exporting data to a wide range of relational databases, including:
- Athena
- DocumentDB
- ElasticSearch
- MongoDB
- MySQL
- Oracle
- Postgres
- Redshift
- Snowflake
- SQL Server
- TeraData
- Any other JDBC-supported database
Use Cases of TO DB
Below are the most common scenarios where TO DB is used:
Export Transformed Data for Analytics and Reporting
After performing transformations in Syntasa, you may need to send the cleaned and enriched data to a reporting or analytics database such as Snowflake, BigQuery, or Postgres. This allows business intelligence tools like Tableau or Power BI to work with the most up-to-date data.
Sync Operational Data to External Databases
You may want to sync updated customer profiles, transactions, or activity logs from within Syntasa into an operational database used by business systems. This ensures that external applications always have access to current and accurate data.
Power Third-Party Platforms and Dashboards
TO DB enables you to integrate Syntasa output with third-party platforms or tools that rely on reading from external databases. These might include marketing platforms, personalization engines, or dashboards that expect real-time data in a database table.
Automate ETL/ELT Data Loads into Warehouses
As part of a modern ETL or ELT pipeline, TO DB allows you to automate data loading into a target warehouse table, eliminating the need for manual intervention or custom scripts. This supports efficient and scheduled data movement in production environments.
Key Features Supported by TO DB
Partitioned Data Support
If your input Event Store contains partitioned data, such as partitions by date, the TO DB process can export only the relevant partitions based on the execution date range. This allows for incremental and efficient database updates, without needing to load the full dataset every time.
Column Renaming
You can rename columns before exporting them to the database. For example, if your input has a column named city, you can rename it to town so that the target table reflects the desired schema.
Selective Column Export
You can choose to export only the required columns to the database. This helps reduce unnecessary data transfer and ensures that only relevant data is written to the target table.