In Databricks, this layering is hidden behind a single product. Syntasa composes it from separate AWS and platform pieces, so the seams are visible — and that is where most of the confusion comes from. This article makes the seams explicit.
It also answers the most common question that arises from them: why a table sometimes appears with a single column in a BI tool, even though every query against it returns the correct data.
How the layers fit together
Reading it layer by layer:
Datastore — a Syntasa concept with no AWS equivalent. It is the access and authorization boundary: who can reach which data, across the dev / prod / snapshot environments. It is not a catalog and it does not store schema.
Dataset — one logical table in Syntasa. Managed means Syntasa created and maintains its catalog entry; external means it points at a table someone else created. That distinction governs the whole lifecycle and is covered further down.
Glue Data Catalog — the Hive-compatible metastore. Holds the database and the table entry: name, storage location, format, and a column list. Every consumer discovers tables here.
Spark — reaches Glue through
spark_catalog. This is what a notebook or an App step query.Athena — reads the same Glue catalog to find the table, then resolves a Delta table's schema natively from
_delta_logon engine v3.Superset and Tableau — sit on top of Athena; Superset's SQL Lab connects through
pyathena. What they get from a query is what Athena resolved, not what Glue holds. A BI tool's field picker is a separate matter — see below.S3 — the data itself. For a Delta table the directory also contains
_delta_log, the transaction log.
Two sources of schema truth
This is the single most useful thing to understand about the model, and it explains the "table shows one column" behaviour directly.
A Delta table has its schema recorded in two places:
_delta_login S3 — authoritative. Spark and Athena both resolve the schema from here when they run a query.The column list on the Glue entry — used for display and discovery: the Glue console, Lake Formation, and tools that build a field list from a
GetTablecall.
Historically, Spark wrote a placeholder — a single col array<string> column — into the Glue entry when creating a Delta table, and never updated it afterwards.
Who actually sees one column. Anything that reads the Glue column list directly — the Glue console, Lake Formation, BI field pickers — sees a single column. This is most visible in BI tools whose field picker reads the catalog directly, such as Tableau. Athena resolves Delta natively on engine v3, so anything that queries through Athena gets the real columns.
Note that a single BI tool can sit on both paths at once: its field picker reads the catalog entry, while the queries it then runs go through
Athena. That is why a table can list one column in the field picker and still return every column when queried — and why the answer depends on which view of the tool you are looking at, not on the tool itself.
Query correctness is never affected. A table in this state is a stale catalog entry, not a broken table.
How the catalog entry is kept in sync
From 9.1.0.16, Syntasa keeps the Glue entry in step with the schema in the Delta transaction log, so catalog consumers see the real columns.
Sync happens on write — CREATE, ALTER, INSERT, MERGE — so a table's entry is corrected on its next write.
It is best-effort by design: it never blocks or fails a job, and a missed sync corrects itself on the table's next write.
It compares both sides, so a table that is dropped and recreated is re-synced rather than assumed current.
It applies wherever Glue is the metastore, in App steps and notebooks alike, and does nothing on other metastore backends.
Tables that are no longer written to
Because the sync happens on write, a table that is never written again keeps its placeholder entry indefinitely — it will not correct itself.
A one-off backfill is available to refresh those entries in place. If you have read-only or archived tables still showing a single column in a field picker, contact Syntasa Support. No code change or migration is needed.
Delta is not the default format
Unlike Databricks, Syntasa does not currently default the Spark session to Delta. saveAsTable without .format("delta") writes plain Parquet.
If a table is not Delta, none of the above applies to it: there is no _delta_log, and the Glue column list is its only schema. Making Delta the platform default is planned for a 9.2.x release.
Which metastore is in use
Glue is a configured choice per environment, not a product default. The metastore backend is selected by metastoreType in the Infrastructure configuration, and the supported values are:
AWS_GLUEAWS_RDS_POSTGRES,AWS_RDS_MYSQLGOOGLE_SQL_POSTGRES,GOOGLE_SQL_MYSQLAZURE_SQL_POSTGRESLOCAL_POSTGRES
When the value is AWS_GLUE, no JDBC connection details are supplied to Spark. Instead the session is pointed at the AWS Glue Data Catalog client factory, com.amazonaws.glue.catalog.metastore.AWSGlueDataCatalogHiveClientFactory. The property name that carries it differs by execution environment. For the JDBC-backed types, a metastore URL, username and password are resolved and passed through instead.
So "Glue is the metastore" is true of a given environment and should be confirmed against that environment's Infrastructure settings rather than assumed. On a non-Glue backend, the Athena and Glue-console layers in the diagram simply are not present.
What Syntasa creates for a managed dataset
For a Syntasa-managed dataset, the write path maintains the metastore objects automatically. No manual registration is required.
Table registration —
CREATE EXTERNAL TABLE IF NOT EXISTS <database>.<table> ... LOCATION '<path>'is issued as part of the write.Partition discovery — on AWS,
ALTER TABLE <database>.<table> RECOVER PARTITIONSis run so new partitions become visible.MSCK REPAIR TABLEis used on non-AWS providers.
One dataset is one Glue table.
Legacy _athena tables
Earlier releases created a second table alongside each Delta dataset — <table>_athena, pointing at <location>/_symlink_format_manifest/ — so that Athena could read it. Athena engine v3 reads Delta natively, so current releases no longer create them.
Environments upgraded from an older build may still hold leftovers. Syntasa cleans them up rather than maintaining them:
partition repair touches the twin only if it already exists;
deleting a managed dataset drops the twin alongside the main table;
re-registering an external dataset drops any leftover twin explicitly, so it cannot serve stale data.
Existing _athena tables keep working, so there is no migration to run. Treat one in the Glue console as a leftover to be cleaned up, not something Syntasa is maintaining.
What Syntasa removes on delete — and what it does not
Deleting a managed dataset drops the metastore table, any legacy Athena twin, and optionally the underlying data.
Deleting an external dataset does none of that. The delete path returns early, before any metastore or storage operation, and logs that it skipped the drop because the dataset is external.
This is deliberate. An external dataset points at a table and a storage location that Syntasa did not create, and removing the Syntasa-side reference must not destroy someone else's data or catalog entry.
The UI "drop" action reaches the same code path, so it behaves identically.
Managed vs external at a glance
Aspect | Syntasa-managed dataset | External / user-managed dataset |
|---|---|---|
Metastore table creation | Created by the write path ( | the UI, or by you outside Syntasa |
Partition maintenance |
| User's responsibility |
Glue column list kept current | Yes, on each write (9.1.0.16 and later | Set at register time from the supplied schema |
Delete in Syntasa — metastore |
| Skipped entirely — no metastore call is made |
Delete in Syntasa — storage | Data folder deleted when data removal is requested | Skipped entirely — data is left in place |
Who cleans up Glue | Syntasa | Nobody, today — see the limitation below |
Known limitation: an external dataset's Glue entry cannot be removed from the UI
Where you have no direct Glue access, an externally registered table's metastore entry currently has no removal path. Every route behaves the same way:
Delete the dataset in Syntasa — skipped because the dataset is external. No
DROP TABLE, no storage delete.The UI drop action — reaches the same code path, same outcome.
Re-register the external dataset — this is the one UI flow that does issue
DROP TABLE IF EXISTSagainst the entry, but it drops only in order to recreate immediately, so it never leaves the entry removed.The Syntasa API — offers no drop or unregister endpoint.
The skip itself is correct behavior: Syntasa must not delete a catalog entry or data it did not create. What is missing is a deliberate, opt-in way for you to say "also remove the Glue entry" for a dataset you registered. That is planned for a future release. In the meantime, contact Syntasa Support if you need an entry removed.
Service accounts and datastore metadata operations
From 9.2.0, a datastore can carry Locations and an attached service account.
The service account is used for metadata operations only: Validate, Create Databases, and the existing external-dataset register, get-schema and validate flows. Credentials resolve per request to short-lived cloud credentials — AWS STS AssumeRole, GCP impersonation, or Azure OAuth.
Two limits are worth stating precisely, because they are easy to over-read:
The attached service account is never used by policy generation. It changes which identity performs metadata operations; it does not grant or widen access.
Authorization for the added Locations is separate, and additive to the existing dev / prod / snapshot environments. Snapshot behavior is unchanged.