poka.sh

 Docs

Write a parser

tutorial

This tutorial takes you from an empty crate to a file previewing in Poka. It follows one path: a text-based format that fits a shared viewer. By the end you have a poka-<ext> crate that decodes the format to an AST, a manifest, a corpus test, and a preview in the app.

Read SPEC.md alongside this page. It is the normative contract — manifest, WASI ABI, output rules, routing. Where this tutorial and SPEC.md differ, SPEC.md wins.

Before you start

  • The four repos checked out side by side (see the workspace CLAUDE.md).

  • poka-samples beside this repo, so the corpus tests can find real files.

  • The git hooks on, once per clone:

    git config core.hooksPath .githooks

1. Look at real files first

Do not write a parser from a specification alone. Every parser in this repository that was written against a spec and then run over the corpus was wrong in some way, and the ways were not exotic:

Format What the corpus showed
.eso Conditional reporting appends a gating schedule after the frequency (!TimeStep,REPORTSCH). 75 lines in the corpus.
.ctb Not a text file at all — an ASCII header then a zlib stream.
.dwl2 AutoCAD writes <?xml … encoding="UTF-8"> with no closing ?>. A conforming XML parser rejects its own vendor's output.
.xcu LibreOffice writes xsi:nil without declaring xsi. And oor:finalized lives on <node>, not <prop> — all 1,762 occurrences.
.ott 11 of 27 real templates declare the plain document media type.
.blo Two unrelated container families share the extension.
.cnt A Shift-JIS file sitting among the Polish ones, in a format that declares no encoding.

Open several files of your format, by hand, before you write code. Note the things that defeat a naive implementation: the header that is not a column, the encoding nobody declares, the two families behind one extension.

Samples live in the sibling poka-samples repository, flat in sample files/. Record each sample's provenance in docs/sample-inventory.csv: one row per sample, with a list column saying which inventory it belongs to (stub-triage, the hunt for samples of formats that had a hex stub, or bem-inventory, the building-energy-modelling samples) and columns for the extension, filename, format, vendor or application, magic bytes, size, SHA-256, source, source URL and licence. Do not put personal or client data in poka-samples; if the only samples you have carry usernames or client project names, keep them in the crate's own samples/ directory instead; the test helper searches both locations.

2. Check for an existing parser

Check the market before you write anything. Is there a library? Is its licence commercially usable? Is it in Rust? Record what you find in the format's Libraries cell in FILE_FORMATS.csv, whether or not you adopt it. The cell grammar, and the tool that researches many formats at once, are in Research libraries for a format.

3. Scaffold the crate

parsers/poka-<ext>/
  Cargo.toml        [dependencies] poka-preview-proto.workspace = true
  plugin.json
  src/lib.rs        the parser
  src/main.rs       the WASI shim
  tests/samples.rs  the corpus test

There is no per-crate build.rs. The crate depends on poka-preview-proto (core/poka-preview-proto), whose one build script compiles core/proto/preview.proto once for the whole workspace. Declare the module in lib.rs or main.rs:

mod proto {
    pub use poka_preview_proto::preview;
}

then use proto::preview::*;. Do not copy the proto and do not check in a generated poka.preview.rs.

If the format is in the hex-stub backlog, it is a line in parsers/unsupported.toml. Delete that line in the same change, or the new crate's directory and the route collide.

Because the workspace members are a glob (parsers/*), one invalid Cargo.toml under parsers/ breaks every cargo command for everyone. Write the manifest to a temporary file outside parsers/, then move it into place.

4. Split lib and bin

Put the parser in lib.rs and the WASI plumbing in main.rs. This is not cosmetic: integration tests cannot include! a binary crate, so a parser that lives in main.rs cannot be run over the corpus at all.

main.rs is the standard shim and nothing else: decode the PreviewRequest, read /input/<basename>, call build_ast, and write one AST — or one error PreviewResponse — to stdout.

5. Write the module documentation first

Describe the format with a worked example, and, most importantly, the things that defeat a naive implementation. Future readers need the traps, not a restatement of the happy path.

//! Two details make a naive split wrong:
//!
//! * The header's own first field is the record type wrapped in angle
//!   brackets, and it is *not* a column — the data row repeats the type in
//!   that position. Treating it as a column shifts every value by one.

6. Test at two levels

Unit tests in lib.rs pin the grammar against hand-written fixtures. Name them after the behaviour, not the function:

#[test]
fn a_gating_schedule_does_not_leak_into_the_frequency() { … }

Corpus tests in tests/samples.rs run over the real files. Assert properties, not golden output — golden output for 25 files is unmaintainable and tells you nothing when it breaks:

  • Every sample parses, and yields the fields the format guarantees.
  • Counts are internally consistent (a histogram sums to its total).
  • The corpus still exercises the paths you care about.
assert!(dbcs > 0, "no sample exercised double-byte detection");

7. Fill in plugin.json

Set name, viewer, and caps that fit the format. Be honest about memory: an ESO can be 200 MB, a .dwl is 40 bytes. Prefer a shared viewer over bespoke HTML — only go monolithic for a bespoke response type (SvgPreview, ImagePreview, ModelPreview, DwgDrawing). A typical composed parser is 30–200 lines of format logic and zero HTML.

8. Route it

Claim the extension in the manifest. Related formats go through aliases (the same module genuinely handles them) or parsers/aliases.toml, never a second crate. One handler per extension: two different modules claiming one extension is the anti-pattern.

Add the format's row to FILE_FORMATS.csv:

python3 scripts/sync-file-formats.py --apply

9. Build and see it in Poka

cargo build -p poka-<ext> --target wasm32-wasip1 --release

Then install it into the host's extension dir (~/.poka/extensions/) from poka-interfaces:

python3 scripts/install-extensions.py

A fresh clone shows no previews until parsers and viewers are built and installed this way. Open a file of your format in the app.

Checklist

  1. parsers/poka-<ext>/ with Cargo.toml, src/lib.rs, src/main.rs, plugin.json, tests/samples.rs.
  2. Pick the AST that fits; go monolithic only for a bespoke response type.
  3. Bound everything: bytes read, rows or nodes emitted, recursion depth.
  4. Add a sample to poka-samples; run the corpus tests.
  5. If the wasm covers sibling extensions, list them in plugin.json's aliases.
  6. Run python3 scripts/sync-file-formats.py --apply for the FILE_FORMATS.csv row.

Next