INVOKE API ENDPOINT

Production fetch, writes response bodies to the connection's storage zone under a timestamped per-run folder. Honors the stored watermark unless FULL REFRESH, respects a per-endpoint lease unless ALLOW CONCURRENT, and returns a single-row RunSummary.

Category: api-endpointPrivilege: adminDeltaForge extension

Syntax

INVOKE API ENDPOINT <name>
  [USING (<override_key> = <expr>, ...)]
  [FULL REFRESH]
  [ALLOW CONCURRENT]

Description

## Overview INVOKE API ENDPOINT is the production entry point, it is what schedules, pipelines, and operators actually call. It writes real files to the zone's storage root, updates the watermark on success, and records a run log row. ## Behavior - A unique timestamped folder (landed_folder) is created under the zone's storage root. All response bodies for this run land there as files, one per page, named per the write.* options. - The per-endpoint lease prevents two overlapping INVOKE runs on the same endpoint. ALLOW CONCURRENT lifts this for intentional fan-out (e.g. one INVOKE per shard). - Incremental mode consults watermark_column / watermark_value from DESCRIBE, outbound requests carry the watermark as a query param or header per the endpoint's config, and the value is advanced on success. - FULL REFRESH ignores the stored watermark; it does not delete old folders, so recent FULL REFRESH + INVOKE sequences produce multiple landed_folders that a downstream external table may need to union. - The USING (...) clause is the standard runtime-override mechanism. Expressions are evaluated once, before the first HTTP request; subsequent pagination reuses the evaluated values. - On success, the call returns one row shaped RunSummary(run_id, status, pages_fetched, rows_landed, bytes_written, landed_folder, duration_ms, watermark_after). - On failure, the run log captures the HTTP status and error message; the lease is released; FULL REFRESH's watermark is NOT advanced. ## Access Control Requires the admin role. ## Compatibility DeltaForge extension.

Parameters

Name	Type	Description
`name`		Target endpoint as <zone>.<api>.<name>.
`overrides`		Runtime overrides of the form path_param.<k> = <expr>, query_param.<k> = <expr>, or header.<K> = <expr>. Each RHS is a scalar SQL expression evaluated at INVOKE time, it can reference $name script parameters, scalar subqueries, NOW(), and any built-in function. Secret-bearing keys are rejected.
`full_refresh`		When true, ignore the stored watermark and refetch from the beginning. Writes a new timestamped folder; the old one is left in place.
`allow_concurrent`		Bypass the per-endpoint lease so multiple INVOKE calls can run in parallel. Intended for intentional fan-out across shards, not accidental double-invocation.

Examples

-- Incremental fetch (honors stored watermark)
INVOKE API ENDPOINT landing.github.issues;

-- Override a path parameter at runtime, for fan-out across repos
INVOKE API ENDPOINT landing.github.issues
  USING (path_param.repo = 'claude-code')
  ALLOW CONCURRENT;

-- Force a full refetch, driving the 'since' parameter from a scalar subquery
INVOKE API ENDPOINT landing.github.issues
  USING (query_param.since = (SELECT MAX(updated_at) FROM bronze.github_issues))
  FULL REFRESH;

-- Parameterised inside a pipeline script
SET $tenant = 'acme';
INVOKE API ENDPOINT landing.crm.accounts
  USING (path_param.tenant = $tenant, header.X_Request_Id = uuid());

Pitfalls

USING expressions are evaluated ONCE per INVOKE, before the first HTTP call. If the expression is a scalar subquery over a rapidly-changing table, the value is captured at call time and reused for every page of pagination.
ALLOW CONCURRENT disables the safety rail designed to prevent double-run. It is only safe when the USING (...) clause guarantees disjoint fetch surfaces (e.g. one repo per call). Running two bare INVOKEs concurrently will fetch and land overlapping files.
FULL REFRESH does NOT purge prior landed folders. After a FULL REFRESH, the storage zone contains both the pre-refresh and post-refresh data. Downstream external tables either select a single landed_folder or must be prepared to dedupe across runs.
Secret-bearing override keys (authorization, api_key, bearer token header names) are rejected at parse time, the same as in CREATE. Auth must come from the connection's CREDENTIAL, there is no bypass via USING.