BQ Process in UDP running multiple queries for multiple output tables failing

Completed

Comments

2 comments

  • Official comment
    Avatar
    Shahdy Ali Hassan

    bsarath  2 months ago

    If you have to write two outputs from single bq process
    1. Each statement should have full syntax for inserting. Just doing a select statement is not enough for bq processor because it does not know which select statement should be written to which output table.
    2. If BQ Process need to have multiple statements, each statement has to be delimited by --END--
    Comment actions Permalink
  • Avatar
    Shahdy Ali Hassan

     
    bsarath  2 months ago
    I dont see that UDP being used in the App any more.
     
    bsarath  2 months ago
    Can you use that UDP (if you had removed it) again so that I can try running myself for the error
     
     
    Steven Prestridge  2 months ago
    @bsarath I just deleted the UDP node and re-entered it into the app. It is labeled 'Census Single and Quartiles -2'
    Note: That UDP is hardcoded as of right now, but once the node works as expected I am going to update it to be a user defined parameter
     
     
    bsarath  2 months ago
    You mentioned it worked when used as regular BQ Process… is that correct?
     
     
    bsarath  2 months ago
    can you point me to that process where you are writing to two different outputs from single bq process?
     
     
    bsarath  2 months ago
    Atleast one issue I noticed, each statement in bq process need to be delimited by
    --END--
     
     
    Steven Prestridge  2 months ago
    My apologies,
    I ran it in Big Query in GCP and it was able to output 2 different tables, I did not run it specifically in a BQ Process in Syntasa
     
     
    bsarath  2 months ago
    ok
     
    bsarath  2 months ago
    If you have to write two outputs from single bq process
    1. Each statement should have full syntax for inserting. Just doing a select statement is not enough for bq processor because it does not know which select statement should be written to which output table.
    2. If BQ Process need to have multiple statements, each statement has to be delimited by --END--
    (edited)
     
    Steven Prestridge  2 months ago
    Thank you @bsarath I appreciate it, will do that now
     
    Steven Prestridge  2 months ago
    @bsarath Would the insert definitely be the way to go? When I am writing an INSERT INTO, I need a dataset name along with the table name. But in this case there is no dataset in BQ being that I am using temporary tables that are joined from bq public data to get the first output table, so when I am writing the INSERT INTO it is giving me a 'Must be qualified with a dataset'
     
    bsarath  2 months ago
    what is the statement you used in bq console in gcp for inserting the data into table?
     
    bsarath  2 months ago
    INSERT dataset.Warehouse (warehouse, state)
    WITH w AS (
      SELECT ARRAY<STRUCT<warehouse string, state string>>
          [('warehouse #1', 'WA'),
           ('warehouse #2', 'CA'),
           ('warehouse #3', 'WA')] col
    )
    SELECT warehouse, state FROM w, UNNEST(w.col)
     
    Steven Prestridge  2 months ago
    Code works successfully, thank you Sarath!
    0
    Comment actions Permalink

Please sign in to leave a comment.