Triggered a process by using file watcher events and uploading a file to cloud storage. Now how can we get that filename?

Completed

Comments

4 comments

  • Avatar
    Labinot Dvorani

    @..., can you share the link to your app?

    you have two scenarios here 1. if its a non-partitione file then you move on to simply defining the schema for the file in your from file. 
    2.if its a partitioned file it should have a date in the file name you're picking up something like example-2022-04-22 etc and in your from file you would configure to  pick up the file pattern then again your columns would need to be defined in the schema tab. 

    0
    Comment actions Permalink
  • Avatar
    Fahim Ishrak

    Hi Lab, I've sent you the app link,
    For me looks like it's scenario 1. The UDP (pdf_extractor) In the app works by taking in a raw pdf file from google cloud storage as an input parameter so there isn't a need to define a schema. I was wondering if there was a way to directly pass the newly uploaded file (sample2.pdf from the above screenshot) as an input parameter to the UDP (pdf_extractor) automatically using triggers and without manual intervention from us

     

    0
    Comment actions Permalink
  • Avatar
    Mahesh Shenoy

    Shahdy Ali Hassan / Sarath Botlagunta : Can we use file object in this case?

    0
    Comment actions Permalink
  • Avatar
    Labinot Dvorani

    The only way to pass the new file name in the custom parameter is then a variable in the code of the UDP, then once you've defined it create an input parameter that the user can input the file name that contains a regex. You can write a for loop to get the latest version based on the date created in the bucket. 

    0
    Comment actions Permalink

Please sign in to leave a comment.