[0.1 draft] [guide 15 · spec 15 pages]

spec/file-structure

File structure and reading

Layout

A Text Tables file consists of, in order:

text

┌──────────────────────────────┐
│ First line                   │  P texttables=0.1 …        exactly one
├──────────────────────────────┤
│ File block                   │  system lines               zero or more
├──────────────────────────────┤
│ Table 1                      │
│   table block                │  P table … + system lines
│   data section               │  inline rows (if any)
├──────────────────────────────┤
│ Table 2 …                    │  zero or more further tables
└──────────────────────────────┘
  • The first line MUST be line 1 (first-line.md).
  • Every table MUST start with a P table line (tables.md). There is no implicit table.
  • A file with no P table lines is valid only with tables=0 (an empty stub).

Recognition

  1. Detect the encoding from a BOM, if present, and strip the BOM (Encoding).
  2. Take line 1. It MUST be ASCII (first-line-not-ascii, FAIL). Let P be every character before the first space (U+0020).
  3. The file is Text Tables if and only if line 1 contains a space and the characters right after the first space start with texttables=.

A file that does not match is not a Text Tables document — including an empty file and an ExCSV file. A tool MAY still open it as plain CSV/TSV, but that is outside this specification and no Text Tables rule applies to it.

The prefix

  • P MUST be 1–8 characters long.
  • P MUST consist of printable ASCII characters (U+0021–U+007E): no whitespace, no control characters.
  • The first character of P MUST NOT be a letter or a digit.
  • P MUST NOT contain the delimiter or the quote character of the file, or of any table that overrides them.
  • Violations are prefix-invalid (FAIL).

Writers SHOULD use #. Legacy CSV readers skip comment lines by a single character (pandas comment=, R comment.char, DuckDB comment), so a single-table file is readable by them only when they are told the first character of P.

Line classes

Every line of the file, other than records of a data section, is one of:

Class Rule
first line line 1
system line starts with P followed by a space, or is exactly P
comment a system line whose first token after P is --
directive any other system line; its first token is the keyword (lines.md)

A line that is exactly P is an empty comment.

Reading

A parser reads the file top to bottom:

  1. Read the first line (first-line.md).
  2. Read system lines into the file block until the first P table line.
  3. For each P table line: read the following system lines into the table block. The first line that is not a system line starts the table's data section.
  4. A table with inline data reads its data section by count: the header row when header=1, then exactly rows records (tables.md).
  5. After the last record of a table, the next line MUST be a P table line, a comment, or end of file. Directives other than table are not allowed after a data section (directive-after-data, FAIL).
  6. A table with reference= has no inline data: its block ends at the next P table line or end of file. A line that is not a system line there is data-in-reference-table (FAIL).

Count mismatch and recovery

Records are read by count, not by looking at the prefix, so a record may begin with any characters. When the count is wrong:

  • Fewer records than rows — a P table line, or end of file, is reached before rows records were read. The parser MUST treat the P table line as the start of the next table, keep the records read so far, and report rows-mismatch (WARN).
  • More records than rows — after rows records the next line is not a system line. The parser MUST keep reading records until a P table line or end of file and report rows-mismatch (WARN).

Recovery relies on data records never looking like a P table line. Writers MUST choose P so that no record of any data section starts with P followed by a space or equals P. This is what a configurable prefix is for: if the data contains lines starting with # , choose another prefix. A parser that reads such a record by count reports record-starts-with-prefix (WARN).

Data section

  • Records MUST follow the table's effective dialect (delim, quote, null, header).
  • Values MUST NOT contain raw line breaks: one record is one line.
  • With quote=none, values MUST NOT contain the delimiter.
  • An empty line inside a data section is a record with one empty field.
  • A trailing line break after the last line of the file is optional.

Line endings

Lines end with LF or CRLF. Parsers MUST accept both, including mixed.

Encoding

  1. BOM. If the file starts with a BOM, the BOM decides the encoding and is stripped:

    Bytes Encoding
    00 00 FE FF UTF-32BE
    FF FE 00 00 UTF-32LE (checked before UTF-16LE)
    FE FF UTF-16BE
    FF FE UTF-16LE
    EF BB BF UTF-8
  2. No BOM. The file is in an ASCII-compatible encoding (UTF-8, ISO-8859-1, Windows-1251, …). The first line is read as ASCII; encoding= on the first line (default UTF-8) applies to the rest of the file.

  3. UTF-16 and UTF-32 files MUST start with a BOM. Without one, line 1 does not read as P texttables= and the file is not Text Tables.

  4. An encoding= that contradicts the BOM is encoding-mismatch (WARN); the BOM wins. An unknown encoding name is encoding-unknown (WARN); UTF-8 is used. Bytes that do not decode are encoding-invalid-bytes (FAIL).

  5. A table MAY declare its own encoding= for its data (tables.md). System lines are always in the file's encoding.

Writers SHOULD write UTF-8; a writer targeting Excel MAY add a UTF-8 BOM.

Compatibility with CSV/TSV tools

A single-table file with inline data is a valid CSV/TSV file for any tool that skips lines starting with the first character of P. A multi-table file read by such a tool concatenates all tables' data and reads the second table's header row as a data record.

spec/file-structure.md on GitHub