[0.1 draft] [guide 15 · spec 15 pages]

spec/archive

Archive (.txtab.zip) and column storage

Storage per table

Where a table's data lives is declared on its P table line (tables.md):

P table Data
no reference= inline records after the table block
reference=<file> records in an external CSV/TSV file
reference=<dir>/ storage=columns a directory with one .col file per column
  • storage= is rows (default) or columns. Any other value is value-invalid (FAIL).
  • storage=columns MUST have reference= ending in / (storage-columns-without-dir, FAIL).
  • section-size= is allowed only with storage=columns (section-size-without-columns, WARN, ignored).
  • Column storage is designed for archives, but a storage=columns directory MAY also sit next to a .txtab on a file system.

Column storage

text

orders/
├── 00-id.col
├── 01-customer_id.col
└── 02-amount.col
  • One file per physical column (stored and materialized; virtual computed columns have none), named <index>-<name>.col with a zero-padded zero-based index; <index>.col when the column has no name (header=0).
  • Each file holds one value per line, in record order, in the table's encoding; an empty line is an empty field; the table's null marker applies. Values are not quoted and contain no line breaks.
  • Every .col file MUST have exactly rows lines (col-line-count-mismatch, FAIL). A missing file is col-file-missing (FAIL); the number of files MUST equal columns= when declared (col-count-mismatch, FAIL).
  • The directory holds only .col files (or section directories). There is no per-table header file: the table's metadata is its block in the main file.
  • Column names come from the file names and the table's column lines; there is no header row.

Sections

With section-size=N, each column is a directory <index>-<name>/ holding files <start>.col, where <start> is the zero-padded first record of the section.

  • All columns share the same section boundaries (section-boundary-mismatch, FAIL).
  • Sections partition [0, rows); line counts sum to rows (section-partition-error, FAIL).
  • Record n is line n − start of section start = ⌊n / N⌋ · N.
  • Recommended: section-size=10000.

.txtab.zip

A standard ZIP archive holding a main file and, optionally, the external data its tables reference.

text

sales.txtab.zip
├── sales.txtab          ← main file, first entry
├── orders/              ← storage=columns
│   ├── 00-id.col
│   └── 01-amount.col
└── customers.csv        ← reference=customers.csv

Main file

  • The main file MUST be the first central-directory entry (zip-main-not-first, FAIL). Readers MUST NOT scan past it.
  • Its name MUST be the archive name without .zip (sales.txtab.zip → sales.txtab), or data.txtab (zip-main-bad-name, FAIL).
  • It is an ordinary .txtab and describes every table in the archive. There is no separate manifest.
  • reference= in the main file resolves against archive entries, relative to the main file's directory; it MUST NOT escape the archive (reference-escapes-dir, FAIL). If no entry matches, a reader MAY look on the file system next to the archive; if that fails too, reference-not-found (WARN).
  • Entries not referenced by any table MAY be present (attachments); readers ignore them.
  • There is no original-size=: ZIP stores every entry's size and CRC.

Compression and encryption

  • Entries SHOULD use Deflate. Store, Deflate64, BZIP2, LZMA and Zstandard MAY be used; others are zip-unsupported-compression (FAIL). ZIP64 MUST be supported.
  • Archives MAY use standard ZIP encryption (traditional PKZIP or AES). The password comes from the tool (for example --password) and MUST NOT be stored in any Text Tables line or in the comment. An encrypted archive without a password is zip-encrypted (FAIL, operational).

ZIP comment

The archive comment (end-of-central-directory, at most 65535 bytes) lets tools see the schema without extracting.

  • Writers SHOULD write it. It MUST be UTF-8 and MUST be a valid Text Tables prefix: the main file's lines up to and excluding data records — first line, file block, every P table and its block — with the same prefix P.
  • If it does not fit, the writer drops lines, never reorders them:
    1. Always kept: the first line and every P table line.
    2. The remaining budget is filled by priority level, each level added to all blocks at once; the first level that does not fit completely ends the comment, and later levels are not added: file metadata (meta-source, meta-author, meta-created, meta-exported, meta-as-of, meta-license, meta-tool) → column → agg-* → sql-pre-data → sql-post-data → other meta-* → sql-dql → chart / chart-<engine> → note / link. Comments are never included.
    3. If anything was dropped, the line P meta-comment-truncated (a flag, no value) follows the first line.
  • The comment is advisory; the main file wins. A comment that is not UTF-8 (zip-comment-not-utf8), not a valid prefix (zip-comment-not-prefix), or disagrees with the main file beyond truncation (zip-comment-disagree) is a WARN and MUST NOT block reading.
  • txtab peek ARCHIVE reads only the comment; summing data-size= over its P table lines estimates the unpacked size of the data.

Verification

txtab verify ARCHIVE.txtab.zip checks:

  1. the main file is first and correctly named;
  2. every reference= resolves to an entry; column-stored tables have the right files, line counts and sections;
  3. everything txtab verify checks on a .txtab, with rows-mismatch and columns-mismatch escalated to FAIL;
  4. the comment, if present (WARN only).

spec/archive.md on GitHub