File System Utilities – SYNTASA™

`fs` — direct object-store filesystem (S3 / GCS / Azure / HDFS)

The synutils.fs object auto-routes by URI scheme. Schemeless paths (bucket/key/path) are also accepted — the scheme is inferred from synutils.infrastructure.fileSystemPrefix.

Listing methods — output shape at a glance

Same shape across Python and Scala:

Method	What it returns
`list(path, recursive=True)`	Files → basenames (last `/` segment); folder markers → empty string `""`. With `recursive=False`, files → basenames; subdirectories → bucket-relative prefix path (e.g. `"syn-cluster-config/deps/jars/"`)
`ls(path)`	Top-level entries as full URIs — files (`s3://bucket/prefix/file.jar`) + subdirectory prefixes (`s3://bucket/prefix/sub/`)
`listRecursive(path)`	All files recursively as full URIs
`list_file_paths(path)` (alias for `listRecursive`)	All files recursively as full URIs

Methods

Method	Purpose
`ls(path)`	Top-level entries as full URIs
`listRecursive(path)`	All files recursively as full URIs
`list(path, recursive=True)`	Basenames (see table above)
`list_file_paths(path)`	Alias for `listRecursive` (full URIs)
`exists(path)`	True if file or folder-prefix exists
`upload(local, remote)`	Upload single file
`download(remote, local)`	Download single file
`uploadFolder(localDir, remoteDir)`	Recursive upload
`downloadFolder(remoteDir, localDir)`	Recursive download
`copy(src, dest)`	Server-side copy
`move(src, dest, exclude_folders=None, rewrite_subfolders=None)` (Python) `move(src, dest)`(Scala)	Move single file (single-object mode) or copy a directory tree with filtering (tree mode — see below)
`rename(old, new)`	Rename in place (same bucket/container)
`delete(path)`	Delete file or prefix
`mkdir(path)`	Create directory marker
`content(path)`	Read text content
`read(path)`	Read raw bytes (Python: `bytes`; Scala: `Array[Byte]`)
`head(path, maxBytes=65536)`	Read first N bytes as text
`writeText(path, content)`	Write a text file
`stream(path)`	Lazy read stream for large files
`uploadStream(fileObj, path)`	Upload from a file-like object. Returns the provider-native upload response (boto3 `put_object` dict, GCS `Blob`, Azure upload result)
`PREFIX` (property)	Filesystem URI prefix (`s3://`, `gs://`, `abfs://`) inferred from infrastructure
`getBaseDir()`	User's base workspace directory: `/<bucket>/syn-workspace/workspaces/<workspace>`
`getLocalTempDir()`	Local temporary directory (`/tmp` or platform equivalent)

`move(src, dest, exclude_folders=None, rewrite_subfolders=None)` — full signature

Two modes, dispatched automatically:

Single-object move — when src does not end with / and no filter parameters are given. Performs copy + delete on the single object (true move semantic).
Tree mode (legacy copy-with-filter) — when src ends with / or either filter parameter is supplied. Lists every object under src, applies the filters, and copies each remaining object to the corresponding path under dest. Mirrors the legacy notebook S3FileSystem.move / GCSFileSystem.move semantics.

Heads-up: Tree mode does not delete the source. The legacy implementation was effectively a filtered tree-copy despite the move name — this implementation preserves that behaviour for backward compatibility. If you need a true tree-move, follow up with delete() on the source prefix.

Parameter	Purpose
`src`	Source URI. Treat as a directory prefix when it ends with `/` or a filter parameter is supplied.
`dest`	Destination URI.
`exclude_folders` (Python Only)	Folder names whose objects should be left in place (not copied) during a tree-mode call. Each entry is normalised to `"<name>/"` and substring-matched against the relative path — so `exclude_folders=["logs"]` excludes anything whose relative path contains `"logs/"`.
`rewrite_subfolders` (Python Only)	`{old: new}` substring replacements applied to the destination path via `str.replace`.

# Single-object move (true move: copy + delete) 
synutils.fs.move("s3://bucket/a/file.csv", "s3://bucket/b/file.csv") 

# Tree copy with exclusions (legacy behaviour — does NOT delete source) 
synutils.fs.move( 
  "s3://bucket/src/", "s3://bucket/dest/", 
  exclude_folders=["logs", "temp"], 
) 

# Tree copy with subfolder rename 
synutils.fs.move( 
  "s3://bucket/src/", "s3://bucket/dest/", 
  rewrite_subfolders={"old_dir": "new_dir"}, 
)

Legacy aliases (Python only)

These names still work — they delegate to their canonical counterpart. Use the canonical name in new code.

Canonical	Legacy name
`upload(local, remote)`	`put(local, remote)`
`delete(path)`	`rm(path)`
`move(src, dest)`	`mv(src, dest)`
`exists(path)`	`exist(path)`
`exists(path)`	`is_exists(path)`
`mkdir(path)`	`create_folder(path)`
`uploadFolder(src, dest)`	`upload_folder(src, dest)`
`downloadFolder(src, dest)`	`download_folder(src, dest)`
`uploadStream(file, path)`	`upload_stream(file, path)`

Examples

Python

synutils.fs.upload("local.csv", "gs://my-bucket/remote.csv")
print(synutils.fs.exists("gs://my-bucket/remote.csv"))

# Listing — three shapes
synutils.fs.list("gs://my-bucket/data/")           # basenames (legacy AWS shape)
synutils.fs.ls("gs://my-bucket/data/")             # full URIs (top level)
synutils.fs.listRecursive("gs://my-bucket/data/")  # full URIs (recursive)

# Read raw bytes
data: bytes = synutils.fs.read("gs://my-bucket/config.bin")

# Inspect environment-derived properties
print(synutils.fs.PREFIX)            # "gs://"
print(synutils.fs.getBaseDir())      # "/my-bucket/syn-workspace/workspaces/my-ws"
print(synutils.fs.getLocalTempDir()) # "/tmp"

# upload_stream — capture the response (e.g. ETag / VersionId)
with open("/tmp/data.csv", "rb") as f:
    resp = synutils.fs.upload_stream(f, "s3://my-bucket/data.csv")
print(resp["ETag"])

Scala

synutils.fs.upload("local.csv", "gs://my-bucket/remote.csv")
println(synutils.fs.exists("gs://my-bucket/remote.csv"))

// Listing — three shapes
synutils.fs.list("gs://my-bucket/data/").foreach(println)            // basenames
synutils.fs.ls("gs://my-bucket/data/").foreach(println)              // full URIs (top level)
synutils.fs.listRecursive("gs://my-bucket/data/").foreach(println)   // full URIs (recursive)

// Read raw bytes
val data: Array[Byte] = synutils.fs.read("gs://my-bucket/config.bin")

// Inspect environment-derived properties
println(synutils.fs.PREFIX)            // "gs://"
println(synutils.fs.getBaseDir())      // "/my-bucket/syn-workspace/workspaces/my-ws"
println(synutils.fs.getLocalTempDir()) // "/tmp"

{[{category.name}]}

fs — direct object-store filesystem (S3 / GCS / Azure / HDFS)