Process modes provide the app with instructions on how to handle the processing of incoming data, such as dropping any data or only processing new data. Below are the options currently available.
Process Mode | Description | Recommendation |
---|---|---|
Drop & Replace | This mode permanently deletes the target table (if exists) and creates a new table. | Use only in Development workflows where new schemas are being constructed and concepts being tested, or in Production when it is clear that a full re-process is necessary. |
Replace Date Range | This mode permanently deletes only the partitions relevant to the date range and re-creates. | Use for Development workflows where schema structure does not change and data outside of processing date range should remain, and Production workflow manual processing in cases where data needs to be completely replaced for the selected processing period of time. |
Add New & Replace Modified | This mode uses source data, e.g. raw files, input datasets, not processed by the respective app pipeline for the provided data range. Also, replace data that has been processed previously, but the modified date has changed since running for the given date range. | Use for most scheduled Production jobs. |
Add New Only | This mode uses source data, e.g. raw files, input datasets, not processed by the respective app pipeline for the provided data range only. | Use for scheduled Production jobs where it is guaranteed that previously processed source data will not change. |