How can use a table generated by one Syntasa instance in another one

Completed

Follow

Shawn Zargham

February 22, 2022 12:28

We have a table that is generated by an active pipeline in one Syntasa instance and want to use it as an input in another active pipeline. What's the best way of doing this?

0

• Official comment

  

  Pradeepraj Chandrasekaran February 22, 2022 21:57 (Edited February 22, 2022 21:57)

  Please use a FromBQ processor in syntasa.

  Steps (Internally performed by FromBQ Process)
  1. Bigquery table has to be exported to cloud storage (in source region(USA)).
  2. Transfered to the bucket in the (dest region (UK)) 
  3. Imported into bq table from cloud storage. 

  Prerequisites:
  1. storage bucket in the source region (US)
  2. storage bucket in the destination region (UK).

  Ref: https://cloud.google.com/bigquery/docs/move-datasets.

  Bigquery also has transfer service. I think it supports cross location loading 

  Comment actions Permalink
• 

  Labinot Dvorani February 22, 2022 18:19

  Shawn Zargham Can i confirm are you loading the data to BQ from your first pipeline in syntasa? 

  0

  Comment actions Permalink
• 

  Shawn Zargham February 22, 2022 18:39

  We want to load from the hive output of the first pipeline that is in US to an app in a UK instance. If the two apps were in the same instance this would work very easily, but not when they are in different instances

  0

  Comment actions Permalink
• 

  Labinot Dvorani February 22, 2022 19:57

  The only way that you are able to do cross project between syntasa instances is loading to Big Query. 
  As far as I know there is no mechanism in our current architecture to access event stores between different instances.
  Sarath Botlagunta  - do you have any recommendations?

  0

  Comment actions Permalink
• 

  Sarath Botlagunta February 22, 2022 20:06

  Yes. BigQuery is the easy option.
  Other option would be to use GCS/S3 connection to that source instance and use From File Process to load the data to the target instance and then continue using it.

   

  0

  Comment actions Permalink
• 

  Shawn Zargham February 22, 2022 21:29

  One instance is in US and the other in UK. Last time we tried doing a BQ copy we ran into an issue, but that was with another company. We can try BQ copy and let you know.

  0

  Comment actions Permalink
• 

  Labinot Dvorani February 22, 2022 21:53 (Edited February 23, 2022 10:23)

  Shawn Zargham / Eric -

  what version are the envs? There is a restriction in BQ for querying cross regions.. example if you have a WITH statement that's querying 2 data sets in two regions that fails, as it doesn't allow it, the same applies for reading from 1 region and writing to another region.
  
  extract from slack:
  
  

  Sarath Botlagunta: From BQ should work across cross locations (UK and US) as long as the connection json being used has permissions. Not sure if it is broken recently BQ Processor was not working from the beginning and we added a fix in 6.1.0 to accept @datasetLocation parameter.

   

  SMA-5466

  0

  Comment actions Permalink

Please sign in to leave a comment.