A land seismic crew drops one folder of SEG-D field records per acquisition day, and a scheduled loader reads each day's drop in place and appends it to a curated Delta table the QC desk works from. Six records, 138 traces across two acquisition days. DISCOVER registers the landing folder, recognising SEG-D from its general header rather than its extension, and passes include_samples = false through so a QC pass over a whole day never touches the sample payload. The loader keys on the record's own file name, so the demo runs the 11 March load twice and asserts the trace count did not move. Two faults are planted in the second day's drop and the QC queries find both: one record captured 18 of the 24 channels because a receiver line went down, and one came off the crew's second truck still configured for a 2 ms sample interval on channel set 2. SEG-D headers are binary-coded decimal, so file number 1041 is stored as the bytes 0x10 0x41 and reads as 4161 to anything treating them as an integer, which one query asserts against directly.
-- ============================================================================
-- SEG-D Field Record QC - Setup Script
-- ============================================================================
-- A land seismic crew drops one folder of SEG-D field records per acquisition
-- day. The processing centre runs a scheduled loader against that folder: each
-- day's drop is read in place and appended to a curated Delta table that the
-- QC desk and the processing geophysicists both work from.
--
-- This file declares the catalog objects only. The loads themselves live in
-- queries.sql, where the assertions between them are what prove the load is
-- incremental rather than a full reload.
--
-- 1. field_records external, registered by DISCOVER over the landing
-- folder; headers only, one row per trace
-- 2. record_1043 external, one record with its sample payload
-- 3. trace_inventory DELTA, the curated target the loader appends to
--
-- SEG-D headers are binary-coded decimal, which is why none of this can be
-- done with a text tool: the file number 1041 is stored as the two bytes
-- 0x10 0x41 and reads as 4161 to anything that treats them as an integer.
-- ============================================================================
-- ----------------------------------------------------------------------------
-- STEP 1: Zone and schema
-- ----------------------------------------------------------------------------
CREATE ZONE IF NOT EXISTS {{zone_name}} TYPE EXTERNAL
COMMENT 'External tables - demo datasets and file-backed data';
CREATE SCHEMA IF NOT EXISTS {{zone_name}}.seismic_acquisition
COMMENT 'Field seismic acquisition: SEG-D records read in place, curated into Delta';
-- ----------------------------------------------------------------------------
-- STEP 2: Register the landing folder with DISCOVER
-- ----------------------------------------------------------------------------
-- DISCOVER reads the bytes at the path, recognises SEG-D from its general
-- header rather than from the file extension, and registers the external
-- table. Nothing about the format is written by hand here.
--
-- Two options ride along. FILE_METADATA adds df_file_name and df_row_number,
-- and df_file_name is what the incremental loader keys on: it is the record's
-- identity, so a file already loaded is a file the loader can skip.
--
-- include_samples = 'false' skips sample decoding. That is what makes a QC
-- pass over a whole day's acquisition cheap, and it is also what lets one
-- table span records of different geometry: the samples column is a
-- fixed-size list whose width is the record's sample count, so a 512-sample
-- record and a 1024-sample record cannot share a schema while it is present.
--
-- The DROP first is what makes this file re-runnable: DISCOVER in EXECUTE
-- mode refuses to register a name the catalog already holds. Without WITH
-- FILES, so the landed records themselves are untouched.
-- ----------------------------------------------------------------------------
DROP EXTERNAL TABLE IF EXISTS {{zone_name}}.seismic_acquisition.field_records;
DISCOVER {{zone_name}}.seismic_acquisition.field_records
PATH '{{data_subdir}}/landing'
WITH (
FILE_METADATA = true,
include_samples = 'false'
);
-- ----------------------------------------------------------------------------
-- STEP 3: One record with its sample payload
-- ----------------------------------------------------------------------------
-- The same command pointed at a single file rather than a folder, with sample
-- decoding left on. The samples arrive as one fixed-size array per trace, so
-- a trace stays a trace instead of becoming 512 columns.
-- ----------------------------------------------------------------------------
DROP EXTERNAL TABLE IF EXISTS {{zone_name}}.seismic_acquisition.record_1043;
-- (excerpt; run the demo for the full script)## What this demo does A land seismic crew drops one folder of SEG-D field records per acquisition day, and a scheduled loader reads each day's drop in place and appends it to a curated Delta table the QC desk works from. Six records, 138 traces across two acquisition days. DISCOVER registers the landing folder, recognising SEG-D from its general header rather than its extension, and passes include_samples = false through so a QC pass over a whole day never touches the sample payload. The loader keys on the record's own file name, so the demo runs the 11 March load twice and asserts the trace count did not move. Two faults are planted in the second day's drop and the QC queries find both: one record captured 18 of the 24 channels because a receiver line went down, and one came off the crew's second truck still configured for a 2 ms sample interval on channel set 2. SEG-D headers are binary-coded decimal, so file number 1041 is stored as the bytes 0x10 0x41 and reads as 4161 to anything treating them as an integer, which one query asserts against directly. ## What it creates - `{{zone_name}}.seismic_acquisition.field_records` (external table) - `{{zone_name}}.seismic_acquisition.record_1043` (external table) - `{{zone_name}}.seismic_acquisition.trace_inventory` (delta table) ## Running it The demo ships `setup.sql`, `queries.sql` and `cleanup.sql` under `demos/subsurface/segd-field-record-qc`. The queries carry their own `ASSERT`s, so a run that completes has verified its own numbers. Data: 6 files, 0.2 MB.