Creating a File Watcher Event

Imagine a scenario where your data pipeline relies on fresh files arriving in a specific directory. A File Watcher Event is the perfect tool for this situation. You can configure it to monitor a chosen directory at a user-defined interval. Whenever a new file matching a specified pattern lands in that directory, the File Watcher Event springs into action.

This triggers downstream jobs within your Syntasa workflow, ensuring your data pipeline stays up-to-date and processes new information as soon as it becomes available.

Here's a breakdown of the key functionalities of File Watcher Events:

• Monitor Specific Directories: Choose the exact directory you want to keep an eye on.
• Define File Patterns: Specify a pattern (e.g., *.csv) to identify the type of files you're interested in.
• Trigger Downstream Jobs: When a new matching file arrives, your designated jobs within the workflow automatically initiate.
• Active/Inactive Control: Maintain flexibility by marking events as "Active" or "Inactive" based on your current needs.

Creating a New File Watcher Event

Follow this step-by-step guide to create a new File Watcher Event:

1. Click on the hamburger menu and select "Events" under Resources.
2. Click the "+" icon. This will take you to the "Create User-Defined Event" screen.
3. In the "Event Type" field, select "File Watcher."
4. Fill out the fields according to your requirements. Details for each field are provided below.
5. Once you've filled in all the information, click "Save."

Configurable Fields

Let's break down the purpose and function of each field available on the 'Create User Defined Event' screen when 'File Watcher' is chosen as the event type:

• Name: This field allows you to assign a name to the file watcher. This name helps identify the file watcher easily and understand its purpose.
• Description: In this field, you can provide a description of the file watcher. This description can explain what the file watcher does and why it was created.
• Tags: You can use tags to categorize the file watcher. Tags help filter and find specific file watchers. To know more about this, please refer to this article.
• Type: This field lets you select the event type. For this example, we'll choose 'File Watcher' since we are setting up a 'File Watcher' event. This option is pre-selected by default.
• Connection: Choose an existing connection that directs to the cloud storage location (like an S3 bucket or GCS bucket) where the directory to be monitored is located. Currently, events support connections of types including GCS, S3, AWS S3, GCP GCS, FTP, SFTP, and ONPREM (HDFS).
• Poll Interval (Minutes): This setting determines how often the event should check for new files, with a value of 1 indicating a check every minute.
  

  Note: It's recommended to start with 30 to 40-minute intervals to balance performance and cost efficiency, as frequent polling may increase costs with your cloud provider. Note that there is an associated cost with events, which is influenced by the data volumes in the specified connection location.

• File Path: This field specifies the file's location that the file watcher will monitor for changes.
• File Pattern: This field allows you to use wildcards to specify a pattern of files to watch. For instance, *.txt will watch for any file that ends with the .txt extension.
• Active/Inactive: When the toggle is set to active (usually indicated by a green button or checkbox), the file watcher monitors the specified file path and file pattern for changes.

  When the toggle is set to inactive (usually indicated by a red button or unchecked checkbox), the file watcher is essentially paused. It won't monitor the file for changes and won't trigger any events. This is useful when you want to temporarily stop the file watcher without having to delete the entire configuration.

Creating an Event by Copying Existing Event

Syntasa provides an option to create a new event by copying an existing event, whether it's a File Watcher or Adobe Feed Watcher. Here’s how to copy an existing event:

1. Click on the hamburger menu and select “Events” under “Resources”.
2. Click the “+” icon() to go to the “Create User-Defined Event” screen.
3. Enable the toggle for “Copy User Defined Event”. A new field labeled “Source” will appear, displaying a list of existing events.
4. Select an existing event from the list. Once chosen, all the fields under the “Type” and “Adobe Feed Details”/“File Details” sections will be populated with the settings from the selected event.
5. Customize the new event by filling out other fields like “Name”, “Description”, and any other necessary details according to your requirements.
6. Once you’ve entered all the necessary information, click “Save”. A new event will be created with the settings copied from the existing event.

Following these steps will create a new event in Syntasa that inherits the core configuration from an existing event, saving you time and ensuring consistency. You can then customize the new event further to fit your specific requirements.