GENERATE MANIFEST

Generates a symlink format manifest file listing all active data files in a Delta table for use by external query engines.

Category: maintenanceDeltaForge extension

Syntax

GENERATE MANIFEST FOR <table>

Description

## Overview GENERATE MANIFEST creates manifest files that list the absolute paths of all active data files in a Delta table. These manifest files enable external query engines that cannot read the Delta transaction log natively (such as Presto, Trino, and Athena) to access the table's data through standard Parquet file readers. ## How It Works 1. **Snapshot read**: Loads the current table snapshot to obtain the list of all active data files. 2. **Path resolution**: Resolves each file's relative path to an absolute storage path. 3. **Manifest generation**: Writes the file listing to `_symlink_format_manifest/manifest` in the table's root directory. For partitioned tables, a separate manifest is created in each partition directory. 4. **Report**: Returns metrics including the number of manifests generated and files listed. ## Manifest Location For an unpartitioned table at `/data/my_table`, the manifest is written to: ``` /data/my_table/_symlink_format_manifest/manifest ``` For a partitioned table, manifests are written per partition: ``` /data/my_table/_symlink_format_manifest/date=2024-01-01/manifest /data/my_table/_symlink_format_manifest/date=2024-01-02/manifest ``` ## Manifest Invalidation Manifest files are static snapshots. They are not automatically updated when the table changes. After any DML operation (INSERT, UPDATE, DELETE, MERGE) or maintenance operation (OPTIMIZE, VACUUM), regenerate the manifest to reflect the current table state. Stale manifests may cause external engines to read deleted files or miss newly added files. ## Result Set Returns a result set with the following properties: | property | value | |----------|-------| | table | Table name | | mode | Manifest mode (symlink_format_manifest) | | version | Table version at manifest generation time | | manifests_generated | Number of manifest files created | | files_listed | Total number of data file paths in the manifests | | execution_time_ms | Time to generate the manifests | ## Access Control | Privilege | Object | Notes | |-----------|--------|-------| | read | Table | Required to read the table snapshot. | | write | Table storage | Required to write manifest files. | ## Compatibility GENERATE MANIFEST is a DeltaForge extension implementing the Delta Lake symlink format manifest protocol.

Parameters

Name	Type	Description
`table`		Specifies the name or path of the Delta table for which to generate a manifest. The table must be registered in the session. Fully qualified names (zone.schema.table) are supported.

Examples

-- Generate a manifest for an external query engine
GENERATE MANIFEST FOR TABLE orders;

-- Using the full symlink_format_manifest keyword
GENERATE symlink_format_manifest FOR TABLE warehouse.sales.transactions;

-- Generate manifest after optimizing the table
OPTIMIZE orders;
GENERATE MANIFEST FOR TABLE orders;

Pitfalls

Manifests are point-in-time snapshots. They become stale after any write operation on the table. Always regenerate manifests after DML or maintenance operations.
External engines reading from manifests do not benefit from Delta Lake features such as time travel, schema evolution, or ACID transactions.
For very large partitioned tables, manifest generation can create thousands of manifest files (one per partition), which may be slow to generate and consume additional storage.