How can use a table generated by one Syntasa instance in another one
CompletedWe have a table that is generated by an active pipeline in one Syntasa instance and want to use it as an input in another active pipeline. What's the best way of doing this?
-
Official comment
Please use a FromBQ processor in syntasa.
Steps (Internally performed by FromBQ Process)
1. Bigquery table has to be exported to cloud storage (in source region(USA)).
2. Transfered to the bucket in the (dest region (UK))
3. Imported into bq table from cloud storage.Prerequisites:
1. storage bucket in the source region (US)
2. storage bucket in the destination region (UK).Ref: https://cloud.google.com/bigquery/docs/move-datasets.
Bigquery also has transfer service. I think it supports cross location loading
Comment actions -
Shawn Zargham / Eric -
what version are the envs? There is a restriction in BQ for querying cross regions.. example if you have a WITH statement that's querying 2 data sets in two regions that fails, as it doesn't allow it, the same applies for reading from 1 region and writing to another region.
extract from slack:Sarath Botlagunta: From BQ should work across cross locations (UK and US) as long as the connection json being used has permissions. Not sure if it is broken recently BQ Processor was not working from the beginning and we added a fix in 6.1.0 to accept @datasetLocation parameter.SMA-5466
Please sign in to leave a comment.
Comments
7 comments