Dataset Utilities

synutils is the unified entry point in every notebook — a Python attribute and a Scala REPL alias. Each section below lists user-callable methods with examples in both languages.

 

datasets — dataset registry + Hive table metadata

 

Method

Purpose

get(datasetName)

DataSet object (table name, partition cols, etc.)

create(datasetName, fileFormat=PARQUET)

Register a new dataset

list(eventStoreName)

All datasets under an event store (devDatasets + prodDatasets combined). Each entry has an environment field — filter on that if you only want one env.

 

DataSet object methods: tableName(), getPartitionColumns(), getNonPartitionColumns(), isPartitioned().

Python

 
# Fetch 
ds = synutils.datasets.get("user_events") 
print(ds.tableName(), ds.isPartitioned()) 

# Register a new dataset (default file format is PARQUET) 
from synutils.file_format import FileFormat 
synutils.datasets.create("my_new_dataset") 
synutils.datasets.create("my_avro_dataset", fileFormat=FileFormat.AVRO) 

# List all datasets registered under an event store 
# (devDatasets + prodDatasets combined; filter by 'environment' for one env) 
for d in synutils.datasets.list("TestStore"): 
    print(d["name"], d["environment"], d["database"])
  
# Only PRODUCTION datasets 
prod = [d for d in synutils.datasets.list("TestStore") if d["environment"] == "PRODUCTION"]

Scala

 
// Fetch 
val ds = synutils.datasets.get("user_events") 
println(s"${ds.tableName} ${ds.isPartitioned}") 

// Register a new dataset (default file format is "parquet") 
synutils.datasets.create("my_new_dataset") 
synutils.datasets.create("my_avro_dataset", fileFormat = "avro") 

// List all datasets registered under an event store (dev + prod combined) 
synutils.datasets.list("TestStore").foreach { d => 
  println(s"${d("name")} ${d("environment")} ${d("database")}") 
} 

// Only PRODUCTION datasets 
val prod = synutils.datasets.list("TestStore") 
  .filter(_("environment") == "PRODUCTION")