Generic Input Adapter

Generic Input Adapter is an app that provides the ability to ingest files for Generic Event (time-based) data (e.g. Transactional Credit Card, CRM, Call Center, Enterprise data, etc.) Leveraging Generic data to provide an understanding of a customer's behavior across different systems. 

Prerequisites

The following screens need to be populated before configuring this app:

  1. Infrastructure - All required fields populated with the environment details of your on-premise, Google Cloud Platform, or AWS environment.
  2. Event Store - Where data will reside in your cloud environment or HDFS cluster.
  3. Connection - Where data will be ingested, for processing by SYNTASA.

App creation

  1. Click on the menu icon ( mceclip3.png ) and under Platform select "Workspace" from the sub-menu. 
  2. Then click the green "+" plus sign button, on the top right of your screen. 
  3. Fill in the New App screen.
    1. Name = display name of your new Generic Input Adapter application.
    2. Key = will automatically populate based on the name you enter.
    3. App prefix = will automatically populate based on the name you enter, you are able to change this if you prefer something else.
    4. Template = Choose your app template, in this case,  Generic Input Adapter
    5. Description = Purely informational text field.
    6. Event Store = Drop down where you can choose your pre-configured event store.
    7. Override Icon = Toggle button, (Don't change, unless you would like to use a custom icon).  
  4. Click 'Create'. 

Configure Generic Input Adapter 

  1. Now find your new app and click on it to open it.
  2. Click the lock icon (mceclip0.png) on the top-left to unlock the workflow. 
  3.  From the left side menu, under Stores drag a Connection onto the workflow. 
  4. Click on the new node you've dragged on and from the drop-down select your connection
    1. Regardless of which connection you're using, configuration options are the same.
    2. Now save the changes by clicking tick (mceclip7.png) on the right.  
    3. Select the edge of your connection and drag it to From File and snap it into place. This will enable From File for editing. 
  5. Let's configure From File by clicking on it.
  6. Input is for the configuration of picking up and reading through the files. 
    1. Source Path: Location to the files (e.g. /generic/data/daily_files/)
    2. Source File pattern
      1. For daily files, you may have the date at the end of your file name e.g. file_name.*.tsv. Your file extension may be different from our example.
      2. For static files, then just write the name of your file. 
    3. Text File type is a drop-down, select the option which applies to your data.
      1. Your selection may change the options available under Events, that's okay we will cover this further down. 
    4. Again Compression Type is also a drop-down, select the option which applies to your data if no compression is being used then select None. 
    5. Now select from the drop-down the Event Data Format, we present the different options please use the ones applicable to your data. 
      1. Ours is delimited, so we will use Delimited. 
        1. Select field delimiter, ours is ","
        2. Leave the remaining two as default (Quote Character, Escape Character).
      2. Select Avro
      3. Apache Log
        1.  if Apache Log Format: Common, then leave Regex Pattern with the auto-completed text. 
        2. If  Apache Log Format: Combined, then leave the Regex Pattern with the auto-completed text 
        3. If  Apache Log Format: Custom, then for Regex Pattern you must write your own Regex logic.
    6. If your data contains headers, toggle Contains Headers ON. 
    7. Incremental Load is Toggled ON if your data is not daily toggled this OFF. 
      1. If Incremental Load ON, select your Load Type for your data, this will either be Daily or Hourly.
    8. Date Parsing for this we will need to define the Date Pattern: yyyyMMdd
      1. For Regex Extraction Type: Regex or Index
        1. if Regex .*daily_crm(.*).tsv.gz
          1. Group Number is 1 as we want the first match in the Regex returned. 
      2. If Index (select the start and end position where your Date in the file name starts. (Remember index starts from 1). 
        1. Start index: enter your number here
        2. End Index
    9. Under Date Manipulation leave as default
    10. Now we're finished with Input, from the left side tab select Schema
    11. You will need to locate your column headers, this may be at the top of your data or in a separate file.
    12. if in the same file, as your data open your file and copy your column headers, we recommend you sanity check the headings to ensure they are as you expect by pasting them into a text editor first, and once ready copy again.
      1.  Now click on Add Columns and paste in your headers, make sure you select the appropriate delimiter.
      2. If in a separate file, you have the option to upload your column headers using the Import button or alternatively, you can add them by repeating the previous step. 
      3. Click on the time Source drop-down and select a header from your data. 
      4. Click Time format type and again select from the drop-down, We're using epoch. mceclip0.png
    13. Click on Output and we recommend you update Table Name and Display with one that's in line with your naming convention. 
    14. Now save the changes by clicking tick (mceclip7.png) on the right.  
    15. From the left side menu, under Stores drag an Event Store (mceclip1.png) onto the workflow. 
  7. Click on the Event Store on the workflow and populate the two drop-downs:
    1. Select the Event Store, if you have a long list type the name of your list. 
    2. Select the Dataset, again if you have a long list type the name of your list. 
    3. Click on the tick box on the right to save.  
    4. From the edge of the Event Store click and snap to the edge of the Generic Event Enrich node.
  8. Now click on Generic Event Enrich. 
    1. You are now asked to alias your primary data source, meaning this is what the app will expect when mapping columns in a field titled "Function" and writing enrichment. We recommend you use a simple but consistent naming convention for tables, which everyone will know in your organization.
  9. Click on "Mapping" on the left menu on the new window being displayed. 
  10. Edit Mapping screen. You only need to edit one mapping screen in one Unified Event Enrich process. The mapping screens will be identical and any changes made to one will also show in the other. 
    1. All changes can also be made via Excel by clicking "Export"  to generate and download a CSV.  
    2. The name is a default schema header in the table (non-editable). 
    3. The labels autocomplete, populated with the fields from the input tables. 
    4. Function is where we write our enrichment,
      • pulling in the exact value from the primary input source, by selecting from the drop-down options.
      • Write custom enrichment example case statements, regular expression (regex) etc. mceclip1.png
    5.  Once done with mapping click on Filter on the left-side menu, we don't need to edit unless we identify something during testing that we would like to filter out.  
      • Avoids reading of unnecessary data from the lookup table.
      • Beneficial only when lookup (secondary table) is big.
      • It helps spark perform some optimization while doing product/event enrich.
      • It extracts only values(like Product ID) that are present in the main table from the lookup table and performs the join.
    6. Click on the Output tab on the left, and check:
      1. The Table name, you are able to update this if you have a specific naming convention 
      2. The View name, you are able to update this if you have a specific naming convention 
      3. The Display name on the workflow. 
      4. Under configurations. (These are default settings, please don't modify them without a consult with Syntasa). 
        1. Partition Schema: Daily. 
        2. File Format: Parquet.
        3. Load to Big Query: toggled ON (by default).mceclip2.png
    7. Now save the changes by clicking tick (mceclip7.png) on the right. 
  11. Finally, click on Save and Lock.

Test in development

Now you're ready to test your configuration. 

  1. To test the Generic Input Adapter, click on the nodes one by one while holding the shift key. The nodes will be highlighted in grey with a tick(see the below screenshot) to indicate it's been selected. 
  2. Now click on the"Job" button on the top-right and then click "New Job" from the drop-down. (see the screenshot below). mceclip3.png
  3. You will now be presented with a window for configuring your job, let's populate the below:
    1. The job Name. 
    2. The job Description is informative for the user.
    3. Tag the JOB. 
    4. Process name: auto-populated, non-editable. 
    5.  Runtime is a drop-down, this allows us to choose the type/size of the cluster you want to use for processing.
    6. Process Mode, for the first run, Replace Date Range is sufficient, however, if you're multiple times you may opt for Drop and replace or add new and modified. (It's worth noting Drop and Replace is not advised for production as this will drop your data in Big Query).
    7. Date Range is a drop-down and you have a number of options like Custom, which allows you to select the dates you want to process (From Date / To Date) or alternatives are available like "Last N Days" which will allow you to select a relative date range e.g. last 2 days and if needed you can add an offset. For the purposes of this, we're using Custom so we will have to enter two dates. mceclip0.png
    8. for Preview Record Limit and Default Test File Limit, leave these as default.  
    9. Now click on "Save & Execute" and the job will start. 
    10. Now Click on Activity to expand and show job details on the right side menu.
      1. Activity will have a 1 to indicate a running job. mceclip5.png
      2. Once the job completes, you can click on the output preview. 
        1. The window will have a menu on the left, click on "Preview".

Run job in production

  1. Deploy your development workflow to production. From the development workflow, click the "Deploy" button, and then you will be presented with the below screen for initial deployment.
    1. After the initial deployment, you will be required to create a snapshot name. 
    2. Snapshot is a feature that saves the state of the app, so you can track changes over time.mceclip6.png
  2. Now open the production workflow.
  3. Highlight the nodes you want to include in the job by holding "Shift" and then clicking on the Unified Event Enrich processes. 
  4. Click on the Job button on the top-right menu, and choose "New Job" in the sub-menu. 
    1. Fill in the name, description, and date range. Clicking "Save and Execute", which will start the job. 
  5. Now Click on Activity to expand and show job details on the right side menu.
    1. Activity will have a 1 to indicate a running job. 
    1. Once the job completes, you can click on the output preview. 
      1. The window will have a menu on the left, click on "Preview".