poka.sh

 Docs

Add a format end to end

how-to

This page takes a new file format from "Poka shows nothing" to a preview in both the desktop app and the web app. It covers the whole route across the four repositories and says which delivery path to choose. The detailed craft of writing the parser is in Write a parser; this page is the map around it.

You need the four repositories checked out as siblings under one parent directory (poka-interfaces, poka-parsers, poka-viewers, poka-samples), and a working Rust toolchain with the wasm32-wasip1 target (rustup target add wasm32-wasip1). The scripts in this repository find the siblings through scripts/_poka_paths.py; python3 scripts/_poka_paths.py shows what they resolve on your machine.

1. Check whether the format is already handled

  1. Look the extension up in FILE_FORMATS.csv in poka-parsers, and in the route table web/src/api/formats.json.
  2. If the extension is routed, open a real file in the app first. If it previews correctly, there is nothing to add.
  3. If a module already reads the format under another name, you want an alias, not a new crate: see Route an extension to an existing parser.

2. Choose a delivery path

Path Use it when Where the work lives
A. Host plain-text list The format is plain text or text-based markup and a module would only read the file and show it desktop/src-tauri/src/code_exts.rs in this repository
B. Native host handler The webview can draw the bytes itself (PNG, JPEG, WebP, AVIF, SVG, PDF), or a required Rust crate will not compile to wasm32-wasip1 desktop/src-tauri/src/commands.rs in this repository
C. Dedicated module One format that needs its own decoder and a response only it can draw a new crate in poka-parsers, emitting a PreviewResponse
D. Shared module Several formats decode through one library one crate in poka-parsers, several manifests (or aliases) pointing at the same .wasm
E. Parser plus shared viewer The format is table, tree, text, hex, metadata card, 3D scene or similar a new crate in poka-parsers that emits an *Ast; the viewer already exists in poka-viewers
F. Parser plus viewer, several tabs One file has several natural views (a model beside its tree, one drawing per page) as E, but the parser answers with a MultiView

Most new formats are C, D, E or F. Prefer E: the parser then ships no HTML or CSS and inherits the host theme. Path B is a last resort, because every host-side handler needs a full desktop rebuild and has to be kept in step in both engines.

Path A needs care. The host checks its plain-text list before the module registry, so an extension in CODE_EXTS shadows any module that is added for it later. Use it only for formats that will stay plain text, and never for a format that has its own geometry or structure to draw.

The order in which the host decides is: native handlers, then the plain-text list, then the module registry (How a preview is produced).

3. Write the module (paths C, D, E, F)

Work in poka-parsers and follow Write a parser. In short:

  1. Look at real files from poka-samples before writing any code.
  2. Create parsers/poka-<ext>/ with Cargo.toml, plugin.json, src/lib.rs and a thin src/main.rs. The crate depends on the shared poka-preview-proto crate; it carries no build.rs of its own and no copy of the proto.
  3. Pick the AST that fits (Choosing an AST). Bound everything: bytes read, rows or nodes emitted, recursion depth.
  4. For path E, set "viewer" in plugin.json to the id of an existing shared viewer. For path C, leave viewer out.
  5. For path D, list the extra extensions in aliases, or add a line to parsers/aliases.toml when the format needs its own id or name.
  6. For path F, answer with a PreviewResponse whose type is TYPE_MULTIVIEW. Each tab has a title and one of scene3d, svg, html, dwg (shown as they are) or viewer_input (the bytes of an *Ast of the type the manifest's viewer decodes; both hosts run that viewer once for the tab). A tab with no viewer to run through is dropped, and the host adds a Source tab when the file is text. A change to this mechanism lands in both hosts (extension_host.rs and web/src/engine-core.js); the shared fixture is scripts/fixtures/viewer-tabs.json.
  7. If the format needs a field the schema lacks, change core/proto/preview.proto here first; see Changing the proto below.

Set memory_mb and max_time_ms from the largest real sample, not a guess. They are limits the host enforces.

4. Add a sample to the corpus

Add a real, redistributable file to poka-samples (flat, in sample files/) with a row in its PROVENANCE.csv. Never add personal or client data, or a file you may not redistribute. Follow that repository's CONTRIBUTING.md.

5. Build and install

# build the module (from the poka-parsers repository root, so its .cargo/config.toml applies)
cd ../poka-parsers
cargo build -p poka-<ext> --target wasm32-wasip1 --release

# desktop: link the built wasm into the extensions directory
cd ../poka-interfaces
python3 scripts/install-extensions.py

# web: copy built modules into web/extensions/ for the local web app
python3 scripts/install-web-extensions.py

install-extensions.py installs by symlink, so rebuilding a module is picked up without installing again. Do not set CARGO_TARGET_DIR: each sibling repository pins one shared target directory in its own .cargo/config.toml, and the variable overrides that pin. See Build and run from source for the rest of the setup.

6. Test

From poka-parsers:

cargo test -p poka-<ext>                    # unit tests
python3 scripts/corpus-tests.py poka-<ext>  # the corpus suite
python3 scripts/test-manifests.py           # every plugin.json, including the new one

From this repository:

python3 scripts/test-parsers.py poka-<ext>             # run the installed parser over the corpus
python3 scripts/test-parsers-robustness.py poka-<ext>  # empty, zero, 0xFF, random and truncated input: no panic, no hang

Then open a sample of the format in the desktop app (poka <file>) and in the local web app (cd web && cargo run, then drop the file on http://127.0.0.1:21220). The two must agree. For the engine-level gates (node scripts/test-client-engine.mjs --strict, node scripts/test-parity.mjs) see "Tests" in AGENTS.md.

7. Update the registry and the route table

  1. In poka-parsers, run python3 scripts/sync-file-formats.py --apply. It adds or updates the format's row in FILE_FORMATS.csv, and computes the derived columns. Never edit those by hand.
  2. If the format was a line in parsers/unsupported.toml, delete the line in the same change.
  3. The web route table web/src/api/formats.json is regenerated from the pinned parser and viewer releases, not from your working tree: a new module reaches poka.sh when poka-parsers cuts a release and the pin in web/extension-releases.json is bumped. Run python3 scripts/fetch-extensions.py and python3 scripts/gen_formats_json.py after a bump, and commit formats.json with it.

Routing by content

If the new format shares an extension with others, do not claim the extension a second time: one handler per extension. A shared extension is routed by content. Add a signature to web/src/content-signatures.json (an .xml is routed by its root element with xml_root, and by namespace with xml_ns where the root name is shared), and a case to scripts/fixtures/routing-cases.json. Both hosts read the same signatures. Poka-parsers' scripts/xml-roots.py lists the XML roots in the corpus.

Changing the proto

core/proto/preview.proto in this repository is the canonical copy. poka-parsers and poka-viewers each carry a vendored copy that is kept in step by hand. After you change the canonical file, copy it into both siblings' core/proto/ and rebuild the crates that use it. Never check in a generated poka.preview.rs. scripts/check-interface-parity.py guards the hosts here; scripts/check-proto-drift.py in poka-parsers guards against a second proto copy appearing inside a repository.

Things that catch people out

  • A viewer id must exist. The viewer value must be the id of a crate under poka-viewers/viewers/; test-manifests.py checks it.
  • Licences. No GPL or LGPL code in a wasm module. A native LGPL library may be loaded at run time as a separate library, never linked or bundled. The workspaces run cargo deny.
  • Disabled default features. Many crates pull in C-bound codecs that do not link for wasm32-wasip1. Start with default-features = false and add the minimum.
  • Generated output. A module that generates a mesh (tessellated from STEP, say) returns it as a Scene3DAst, never as file bytes for the host to decode: ModelPreview is retired (poka-parsers#453). A generated raster goes in ImagePreview.data.