poka.sh

 Docs

Add a field to an AST and render it

how-to

The *Ast messages are the only interface between parsers and viewers. Adding a field touches the proto, which lives in three repos, and then the viewer. This page gives the order.

Take a worked example: a TextAst that should carry an encoding string, shown in the header line of the text viewer.

Before you start

  • The change is additive. You add a field with a new number. You never renumber, retype or reuse a number, because parsers and viewers are built and released separately and an older module will meet a newer one (step 7).
  • Work in a worktree in each repo you touch (see CONTRIBUTING.md).

1. Edit the canonical proto

The canonical copy is core/proto/preview.proto in poka-interfaces. Add the field at the end of the message, with the next unused number and a comment that says what it holds and when it is empty:

message TextAst {
  // ... existing fields ...
  FileFacts facts = 8;
  // Character encoding the parser decoded the file from, e.g. "UTF-8" or
  // "Windows-1250 (assumed)". Empty when the format has no notion of one.
  string encoding = 9;
}

2. Copy it to the other two repos

poka-parsers and poka-viewers each carry their own vendored copy at core/proto/preview.proto. They are kept byte-identical by hand: copy the file over.

cp poka-interfaces/core/proto/preview.proto poka-parsers/core/proto/preview.proto
cp poka-interfaces/core/proto/preview.proto poka-viewers/core/proto/preview.proto

Check that nothing else holds a copy. The check script lives in poka-parsers and takes the repo to check with --root:

python3 poka-parsers/scripts/check-proto-drift.py --root poka-interfaces
python3 poka-parsers/scripts/check-proto-drift.py --root poka-parsers
python3 poka-parsers/scripts/check-proto-drift.py --root poka-viewers

What it checks: that the canonical file exists in the repo and is not a broken symlink, and that no other preview.proto exists under the source directories. It does not compare the three copies with each other, so confirm they match yourself (sha256sum on the three files, or cmp).

Nothing generated is checked in. Each repo builds the Rust bindings from the proto at build time; in this repo that is the poka-preview-proto crate. Do not add a poka.preview.rs.

3. Build everything that reads the message

Adding a field changes the generated struct. A place that builds the struct without ..Default::default() stops compiling with "missing fields". In this repo:

cargo build --release --workspace --target wasm32-wasip1

Fix each viewer that builds the message, usually in its tests. In poka-parsers, the pre-push hook runs cargo check --workspace when the proto changes for this reason, and every parser that builds the message needs the same fix.

4. Render the field in the viewer

In viewers/poka-text-viewer/src/main.rs, show it. It is text from the file's point of view, so escape it:

let encoding = if ast.encoding.is_empty() {
    String::new()
} else {
    format!(" · {}", html_escape(&ast.encoding))
};

Handle the empty case on purpose: parsers that have not been updated send the default (an empty string, zero, false), and the page must look as it did before.

5. Test it

Add two tests to the viewer: the field shown when set, and the page unchanged when it is empty. Add the new field to the hostile-string test so a missing html_escape fails. See Test a viewer.

6. Make a parser fill it

In poka-parsers, set the field in the parser that knows the answer. That is the parser repo's work; the tutorial "Write a parser" there describes the flow. A viewer change without a parser that sends the field has nothing to show.

7. What happens when the pieces meet at different versions

The three repos release separately, and poka-interfaces pins the release of each in web/extension-releases.json. Protocol Buffers makes the additive change safe in both directions:

  • A parser that sends the new field to a viewer built before it: the old viewer ignores the field it does not know. The preview looks as it always did.
  • A viewer that expects the field with a parser built before it: the field has its default value, which is why step 4 handles the empty case.

Renumbering or changing a type breaks both, which is why it is not allowed.

8. Host decoders

The desktop host decodes the proto with generated code, so it needs no edit for a field it does not use. The web engine has a hand-written decoder, web/src/proto.js, that reads only the messages the browser puts on the wire: PreviewRequest and PreviewResponse with its arms, and the facts field of an *Ast (astFacts). The *Ast bytes themselves are passed from parser to viewer untouched. So a new field on TextAst needs nothing in proto.js.

A new field in PreviewResponse, in Scene3DAst or Mesh3D (which the hosts still decode themselves, see How poka-3d-viewer draws a model), or in FileFacts (which astFacts reads) may need a change to that decoder. Check it, and make the change in poka-interfaces in both engines, the Rust one and web/src/, in the same PR.

Check the result

  • cargo test --workspace and the wasm build in this repo.
  • The same in poka-parsers for the parser you changed.
  • sha256sum of the three proto copies: one value.
  • python3 build/scripts/check-docs.py --check if you changed a page here.