SET INCREMENTAL CONFIG

Persists incremental loading configuration as Delta table properties for use by INCREMENTAL FILTER.

Category: incrementalDeltaForge extension

Syntax

SET INCREMENTAL CONFIG ON <table>
  COLUMNS (<columns>)
  [DATECOL <col>] [OVERLAP <n> DAYS]
  [FILTER <condition>] [DIALECT MSSQL|MYSQL|POSTGRES]

Description

## Overview SET INCREMENTAL CONFIG persists incremental loading metadata directly in the Delta table's transaction log as table properties. These properties serve as defaults for INCREMENTAL FILTER, eliminating the need to repeat column names, overlap days, and dialect preferences on every invocation. This command is the recommended first step when setting up incremental data loading for a table. Run it once during initial setup, then call INCREMENTAL FILTER with no parameters in your pipeline to use the stored defaults. ## Table Properties All configuration is stored in the `delta.forge.incremental.*` namespace: | Property | Description | Example Value | |----------|-------------|---------------| | `delta.forge.incremental.columns` | Comma-separated list of key columns | `order_id,batch_id` | | `delta.forge.incremental.dateColumn` | Date/timestamp column name | `modified_date` | | `delta.forge.incremental.overlapDays` | Days to subtract from max date | `7` | | `delta.forge.incremental.filter` | Static filter condition | `region = 'us-east'` | | `delta.forge.incremental.dialect` | SQL dialect for output formatting | `MSSQL` | ## Implementation Details The command uses AlterTableBuilder to write table properties into the Delta transaction log. This means: - Configuration survives table rewrites, OPTIMIZE, and VACUUM operations. - Configuration is versioned alongside table data and can be inspected via DESCRIBE HISTORY. - Configuration is visible to any session that opens the table (not session-scoped). - Each SET INCREMENTAL CONFIG creates a new Delta log commit with the updated properties. ## Partial Updates Only the parameters explicitly provided in the SET clause are written. Properties not included in the statement are left unchanged. To fully replace all incremental configuration, run UNSET INCREMENTAL CONFIG first, then SET INCREMENTAL CONFIG with the new values. If no parameters are provided (only the table name), the command returns a message indicating no properties were configured and does not create a log commit. ## Relationship to INCREMENTAL FILTER INCREMENTAL FILTER follows a merge strategy: inline parameters override stored configuration. This allows a pipeline to use stored defaults for most runs while overriding specific parameters (e.g., a different overlap for backfill runs) on demand.

Parameters

Name	Type	Description
`table`		Specifies the target Delta table on which to store incremental configuration. The table must be registered in the current session (via CREATE DELTA TABLE or OPEN DELTA TABLE). Use a fully qualified name (catalog.schema.table) when multiple schemas are in scope.
`incremental_columns`		Key-based incremental columns.
`date_column`		Date/timestamp column.
`overlap_days`		Overlap days for date column.
`filter`		Static filter condition.
`dialect`		SQL dialect.

Examples

-- Configure key-based incremental loading
SET INCREMENTAL CONFIG ON warehouse.orders
    COLUMNS (order_id);

-- Configure combined key and date incremental loading
SET INCREMENTAL CONFIG ON warehouse.orders
    COLUMNS (order_id, batch_id)
    DATECOL modified_date
    OVERLAP 7 DAYS
    DIALECT MSSQL;

-- Configure date-only incremental loading with a static partition filter
SET INCREMENTAL CONFIG ON analytics.events
    COLUMNS (event_id)
    DATECOL event_timestamp
    OVERLAP 2 DAYS
    FILTER region = 'us-east'
    DIALECT POSTGRES;

-- Update configuration for a table (replaces all provided properties)
SET INCREMENTAL CONFIG ON warehouse.orders
    COLUMNS (order_id)
    OVERLAP 14 DAYS;

Pitfalls

Running SET INCREMENTAL CONFIG with only COLUMNS does not clear a previously stored DATECOL or OVERLAP. To remove a specific property, use UNSET INCREMENTAL CONFIG to clear all properties, then re-apply the desired configuration.
The COLUMNS list is stored as a plain comma-separated string. Column names containing commas are not supported.
The table must be registered in the current session. If you receive a 'Table not registered' error, run OPEN DELTA TABLE or CREATE DELTA TABLE first.
Each invocation writes a new commit to the Delta log. Avoid calling SET INCREMENTAL CONFIG in a tight loop.