This article explores the functionalities provided by the synutils.eventstore module in the Syntasa application. These utilities offer functionalities to write data and files to an Event Store using Python.
The synutils.eventstore module provides several methods for writing data from DataFrames to the Event Store. Here's a breakdown of each method and its parameters:
1. writeToEventStore(dataFrame: DataFrame, @outputDateSet1: String, noPartitions=0)
This method writes data from a DataFrame to the Event Store without any data partitioning.
Parameters:
-
dataFrame: The DataFrame containing the data to be written. -
@outputDateSet1: The user-defined name for the output dataset in the Event Store. -
noPartitions(Optional): The number of partitions to create for the data (defaults to 0, meaning no partitioning).
Example:
Python
/**
*
* @param dataFrame
* @param @outputDateSet1 is a user parameter
* @param noPartitions is optional default is 0
*/
writeToEventStore(dataFrame: DataFrame, @outputDateSet1: String)
Example :
writeToEventStore(df,"@outputDateSet1")
Scala
/**
* @param dataFrame
* @param outputDateSet is a user parameter
*/
writeToEventStore(dataFrame: DataFrame, outputDateSet: String)
Example :
writeToEventStore(df,"@outputDateSet1")
2. writeToEventStore(dataFrame: DataFrame, @outputDateSet1: String, noPartitions:Int, partitionedDateColumn:String)
This method writes data from a DataFrame to the Event Store with data partitioning based on a specified date column.
Parameters:
- All parameters are the same as
writeToEventStore(dataFrame: DataFrame, @outputDateSet1: String, noPartitions=0), with the addition of:-
partitionedDateColumn: The name of the column in the DataFrame to be used for partitioning the data.
-
Example:
Python
/**
*
* @param dataFrame
* @param @outputDateSet1 is a user parameter
* @param noPartitions is optional default is 0
* @param @partitionedDateColumn is a user parameter
*/
writeToEventStore(dataFrame: DataFrame, @outputDateSet1: String,noPartitions:Int, partitionedDateColumn:String)
Example :
writeToEventStore(df,"@outputDateSet1",1,"event_partition")
Scala
/**
* @param dataFrame
* @param outputDateSet is a user parameter
* @param numFiles : Number of files per partition. default is 0
* @param partitionedDateColumn is a user parameter
*/
writeToEventStore(dataFrame: DataFrame, outputDateSet1: String,numFiles:Int,partitionedDateColumn:String)
Example:
writeToEventStore(df,"@outputDateSet1",1,"event_partition")
3. writeFileToEventStore(localFilePath: String, eventStoreFilePath: String)
This method writes the contents of a local file to the Event Store.
Parameters:
-
localFilePath: The path to the local file on the system. -
eventStoreFilePath: The desired path for the file within the Event Store.
Example:
Python
/**
*
* @param localFilePath
* @param eventStoreFilePath
*/
writeFileToEventStore(localFilePath: String, eventStoreFilePath: String)
Example:
writeFileToEventStore("filePath","eventstoreFilePath")
Scala
/**
* @param localFilePath
* @param eventStoreFilePath
*/
writeFileToEventStore(localFilePath: String, eventStoreFilePath: String)
Example :
writeFileToEventStore("filePath","eventstoreFilePath")
These utilities provide a convenient way to interact with the Event Store from your Syntasa application using Python. By leveraging these methods, you can efficiently write various data structures and files to the store for further processing or analysis.