File objects Utilities

files — platform file registry

A file object in the platform pairs a base cloud-storage path with a list of files (the object's parameters). synutils.files resolves these to full cloud paths and — for DATA_FILE objects in supported formats — reads them directly into a Spark DataFrame.

Methods

Method

Purpose

get(name)

Full file object dict / Map from the API (cached per name)

getPath(name)

Full cloud paths for all files in this object — returns List[str] / List[String]

getMetadata(name)

Curated subset of metadata with renamed keys

createDataFrame(name, fileName, sep=None, header=True, inferSchema=True)

Read a registered file into a Spark DataFrame. DATA_FILE objects only; supported fileFormat: DELIMITED, JSON, PARQUET, ORC, AVRO

clearCache()

Drop cached responses (force a re-fetch on next call)

createDataFrame parameters

Parameter

Type

Default

Purpose

name

str / String

—

File object name registered in the platform

fileName

str / String

—

Specific file within the object — must match one of the entries in the object's parameters[].name list

sep

str / String

None (Py) / null (Scala) — falls back to the object's API-configured delimiter, then ","

Column separator. Used only for DELIMITED.

header

bool / Boolean

True

First row is a header. Used only for DELIMITED.

inferSchema

bool / Boolean

True

Infer column types from data. Used only for DELIMITED.

Raises:

  • RuntimeError (Py) / SynUtilsException (Scala) if no SparkSession was supplied to init().

  • ValueError (Py) / IllegalArgumentException (Scala) if the object is not a DATA_FILE or its fileFormat is unsupported.

Examples

Python

 
# 1. Inspect a file object 
info = synutils.files.get("daily_report") 
print(info["objectTypeKey"], info["fileFormat"]) 

# 2. Get full cloud paths for every file in the object 
paths = synutils.files.getPath("daily_report") 
# ['gs://my-bucket/reports/sales.csv', 'gs://my-bucket/reports/orders.csv'] 

# 3. Curated metadata subset 
meta = synutils.files.getMetadata("daily_report") 

# 4. Read one file as a DataFrame (uses the object's configured delimiter) 
df = synutils.files.createDataFrame("daily_report", "sales.csv") 
df.show(5) 
 
 # 5. Override CSV options for this read only 
 df = synutils.files.createDataFrame( 
 "daily_report", 
 "sales.tsv", 
 sep="\t", 
 header=False, 
 inferSchema=False, ) 
 
 # 6. JSON / PARQUET / ORC / AVRO — sep / header / inferSchema are ignored 
 events = synutils.files.createDataFrame("event_dump", "events.json") 
 sales = synutils.files.createDataFrame("sales_dump", "2024-01.parquet") 
 orders = synutils.files.createDataFrame("orders_dump", "orders.orc") 
 records = synutils.files.createDataFrame("user_records", "users.avro") 
 # Note: AVRO requires the matching spark-avro JAR loaded into the runtime. 
 
 # 7. Drop cached responses (force a re-fetch on next call) 
 synutils.files.clearCache()

Scala

 
// 1. Inspect a file object val info = synutils.files.get("daily_report") 
println(s"${info("objectTypeKey")} ${info("fileFormat")}") 

// 2. Full cloud paths for every file in the object 
val paths: List[String] = synutils.files.getPath("daily_report") 

// 3. Curated metadata subset 
val meta = synutils.files.getMetadata("daily_report") 
println(meta("type"), meta("fileFormat")) 

// 4. Read one file as a DataFrame (uses the object's configured delimiter) 
val df = synutils.files.createDataFrame("daily_report", "sales.csv") 
df.show(5) 

// 5. Override CSV options for this read only 
val tsv = synutils.files.createDataFrame( 
"daily_report",
 "sales.tsv", 
 sep = "\t", 
 header = false, 
 inferSchema = false ) 

// 6. JSON / PARQUET / ORC / AVRO — sep / header / inferSchema are ignored 
val events = synutils.files.createDataFrame("event_dump", "events.json") 
val sales = synutils.files.createDataFrame("sales_dump", "2024-01.parquet") 
val orders = synutils.files.createDataFrame("orders_dump", "orders.orc") 
val records = synutils.files.createDataFrame("user_records", "users.avro") 

// 7. Drop cached responses 
synutils.files.clearCache()

Tip: getPath() returns paths for all files in the object — useful when you want Spark to read everything in one go via spark.read.csv(synutils.files.getPath("daily_report")). createDataFrame() reads exactly one file at a time, identified by fileName.