Apps - Operations

This section helps to better understand the purpose of the Operations screen and maximize its capabilities.

At a high level, a user can:

Currently, there exist three categories of jobs: test jobs, Deploy jobs, and production jobs. Each job category possesses unique names and is associated with specific processes and logs that track its status. Additionally, individual jobs may offer different options depending on their capabilities.

Deploy

Deploy jobs only track the status of each run and the steps the deploy job processed. To know more about this, please refer to this article.

Test Jobs

Users have the option to re-run the steps defined in the Test jobs, select some processing options detailed below, and view status logs.

  • Double-click the Process Mode element that requires an update.
  • Process Mode - provides an option instructing the app what to do with the table. The options are described below.
    • Drop & Replace - completely deletes the table and re-creates
    • Replace Date Range - deletes only the partitions relevant to the date range and re-creates
    • Add New & Replace Modified - use source data (i.e. raw files, input datasets) not processed by the respective app pipeline for the provided data range. Also, replace data that has been processed previously, but the modified date has changed since running for the given date range.
    • Add New Only - uses source data (i.e. raw files, input datasets) not processed by the respective app pipeline for the provided data range only.
  • Test Record Limit - provides a limit of records to be processed
  • Preview Record Limit - provides a limit of preview records that will be sent back to the app
  • Default Test File Limit - provides a limit of files to be processed

Steps to execute a job:

  • From Development Operations Click on a job
  • Click Execute button
  • Select the Date Range or Relative Range option
  • Select the dates or range of days that should be processed
  • Modify any Limit options, if desired
  • Click Execute
  • Monitor the Jobs section to track job status

Production Jobs

There are three types of scheduling options available: crontab, time-based options (daily, hourly, weekly, and monthly), and triggering.

  • Crontab Scheduling: Crontab scheduling requires the use of a (quartz) cron expression, providing advanced and flexible scheduling capabilities. Users can define specific schedules based on time and date parameters, enabling precise control over job execution intervals, including complex recurring patterns.
  • Time-Based Scheduling: The time-based options (daily, hourly, weekly, monthly) offer a simpler approach. Users can specify a specific time or frequency for job execution. For instance, jobs can be scheduled to run once a day, every hour, once a week, or once a month, without the need for complex expressions.
  • Triggering: The triggering option provides additional flexibility. It allows job chaining based on dependencies between jobs or steps within a job. Jobs can be triggered upon the successful completion of other jobs or specific steps. Additionally, triggering can be based on file delivery, where a job or step can be initiated upon the arrival of specific files. Furthermore, jobs can also be triggered based on crontabs, allowing for scheduling based on predefined cron patterns.

To summarize, crontab scheduling utilizes cron expressions for precise control, time-based options offer straightforward scheduling based on specified intervals, and triggering provides the ability to chain jobs or steps based on dependencies, file delivery, and even custom crontab patterns.


Steps to schedule a job:

  • From the Production Operations screen click on the Jobs tab then click on the desired job
  • Next, click on Configure 
  • Toggle the Schedule button ON
  • Select the relative date range from 1 to 10 is our maximum recommended date range for most data sets
  • Ending on:  options available are current date, yesterday, or Last Available Date 
  • Offset toggle ON/OFF if you want to offset the relative date ranges you entered, in some scenarios this may be useful if you want to introduce a delay in what is processed. 
  • Select the schedule type: 
    • Cron Expression (Quartz): example: 0 2 0-2 ? * * * (This will process jobs three times, 12:02, 1:02, and 2:02 AM, and the timezone. 
    • Hourly: the minute 
    • Daily: hour and the minute, AM /PM and the timezone 
    • Weekly: hour, minute,  day, AM /PM, day of the week, and the timezone
    • Monthly: hour, minute,  day, AM /PM, day of the month, and the timezone
    • Triggering: (a job can have multiple triggers setup)  
      • Adobe File Watcher:  Select the desired event (This requires Events to be set up.)
      • File Watcher: select the desired event (This requires Events to be set up.)
      • Job: select a scheduled job
      • Process: select a single process in a scheduled job
      • Time: this uses a cron expression and a timezone
  • Once complete click SAVE.