poka.sh

Docs

Poka parsers scorecard

Generated by scripts/scorecard.py. Do not edit by hand: python3 scripts/scorecard.py rewrites it, and python3 scripts/scorecard.py --check fails when it is stale.

Generated file. Do not edit it by hand. Run python3 scripts/scorecard.py to rewrite it; --check (pre-push) fails when this copy is stale.

The README headline counts manifests. This page says what sits behind them. It reads only committed files in this repo, so it is the same on every machine: no corpus, no network, no clock. The tier rules are in section 7, with what each tier does and does not prove.

Every row has a basis. measured: counted from files in this tree. estimated: inferred from them; the row says what from, and it is not a claim about the parsers' behaviour.

1. Format census

Counted by scripts/format-stats.py, the single source for the format count, except where a row says otherwise.

Figure Count Basis
Directories under parsers/ 892 measured
Of those, without a plugin.json (helper libraries, not parsers) 13 measured
Manifests (crate plugin.json plus routes) 1,045 measured
crate manifests 879 measured
routes in parsers/aliases.toml 166 measured
stubs in parsers/unsupported.toml 0 measured
Distinct manifest ids 1,045 measured
Unique file_extension values (disabled manifests included) 1,016 measured
Unique alias extension entries 649 measured
Extensions claimed by more than one manifest 0 measured
Disabled manifests 32 measured
Enabled manifests 1,013 measured
Crate manifests that are not valid JSON 0 measured

2. Depth tiers

Every manifest is in exactly one tier. Counts are measured; what a tier means is the estimate, and section 7 says how far it can be trusted.

Tier Manifests Share Basis
T1 Parser with a shared viewer 558 53.4% measured
T2 Standalone parser 20 1.9% measured
T3 Metadata card 272 26.0% measured
T4 Alias-only route 163 15.6% measured
T5 Hex stub (backlog) 0 0.0% measured
T6 Disabled 32 3.1% measured
Total 1,045 100.0% measured

Hex stubs split three ways: 0 in parsers/unsupported.toml, 0 other routes that run poka-hex, and 0 crates of their own whose viewer is poka-hex-viewer (measured).

Alias-only routes by the module they run (measured, top 10):

Module Routes
poka-code 80
poka-xml 13
poka-pe 12
poka-text 11
poka-ini 6
poka-cfb 5
poka-compressed 5
poka-pem 3
poka-soe 3
poka-doc 2

3. What the headline counts

scripts/kpi-stats.py counts every enabled manifest as supported except the unsupported.toml stubs and routes to poka-hex. That includes crates that only hex-dump a file, because they have a source file of their own. The scorecard moves those to T5.

Figure Count Basis
Counted as supported by kpi-stats (README headline) 1,013 estimated: the kpi-stats rule applied to the manifests (T1 to T4 plus own-crate hex shells)
of which own-crate hex shells 0 measured
Supported with hex shells removed (T1 to T4) 1,013 measured
Own crates that read something (T1 to T3, upper bound) 850 estimated: the viewer a manifest names, not the code behind it
Hex-dump backlog (T5) 0 measured

4. Viewer usage

Enabled manifests by the viewer they name (measured). A standalone parser names none.

Viewer Manifests Crates Routes
poka-tree-viewer 318 281 37
poka-metadata-card-viewer 273 272 1
poka-table-viewer 113 99 14
poka-code-viewer 81 1 80
poka-text-viewer 71 50 21
poka-image-with-meta-viewer 47 45 2
poka-3d-viewer 41 41 0
(none: standalone) 26 20 6
poka-vector-viewer 21 21 0
poka-workbook-viewer 9 8 1
poka-slide-viewer 4 3 1
poka-chart-viewer 3 3 0
poka-ebook-viewer 2 2 0
poka-diff-viewer 1 1 0
poka-font-viewer 1 1 0
poka-pdf-viewer 1 1 0
poka-svg-viewer 1 1 0

5. Test presence by tier

Crates under parsers/ that have each file or marker (measured). Presence is not a pass: tests/samples.rs needs the poka-samples corpus, so it runs locally (scripts/corpus-tests.py), never in CI. Alias-only routes have no crate, so they are not in this table.

Tier Crates src/lib.rs Unit tests tests/samples.rs tests/scaling.rs tests/bench.rs Quarantined
T1 Parser with a shared viewer 558 95 231 92 9 35 0
T2 Standalone parser 20 1 7 1 0 0 0
T3 Metadata card 272 17 35 16 0 9 0
T6 Disabled 29 1 3 1 0 1 0
Helper crates (no plugin.json) 13 11 11 2 0 0 0
Total 892 125 287 112 9 45 0

Unit tests means a #[cfg(test)] module under src/. Quarantined crates are listed in build/unit-test-quarantine.txt and skipped by the unit-test gate (0 listed).

6. Never-crash lint adoption

The root Cargo.toml has no [workspace.lints.clippy] table that denies panicking constructs, and no crate sets [lints] workspace = true, so no crate is held to the never-crash rule by a lint (measured). The rule is written in docs/how-to/make-a-parser-fast.md, "Never panic", and is exercised by the robustness sweep (scripts/quality-gates.py), not enforced by clippy. This section will report adoption by tier once there is a lint table and crates opt in.

7. How the tiers are derived

First match wins, in this order: disabled, hex stub, alias-only route, standalone, metadata card, shared viewer. Everything comes from plugin.json, parsers/aliases.toml, parsers/unsupported.toml and the module a route names; no source file is parsed for meaning.

T1 Parser with a shared viewer. A plugin.json crate, not disabled, whose viewer is a shared viewer other than poka-hex-viewer and poka-metadata-card-viewer.

  • Shows: The crate has its own source and emits a typed AST (tree, table, text, image, 3D, vector, workbook and so on) for a viewer poka-viewers ships.
  • Does not show: How much of the format it reads. A tree of a few top-level keys and a full decoder of a binary container both land here. Depth is shown only by a corpus test (section 5) and the module's own documentation.

T2 Standalone parser. A plugin.json crate, not disabled, with no viewer. SPEC.md section 2: it writes a complete PreviewResponse itself (3D scenes, PDF, media, archives).

  • Shows: The crate has its own source and renders its own result.
  • Does not show: That the result is complete, or that it is not a dump in another wrapper. Nothing in the manifest says what the response holds.

T3 Metadata card. A plugin.json crate, not disabled, whose viewer is poka-metadata-card-viewer. This is the tier of the generic-container and identification shells.

  • Shows: The crate reads the file far enough to fill a titled list of key and value entries.
  • Does not show: That there is more than identification. A card can carry a rich header or a single magic-number line; the manifest does not tell them apart.

T4 Alias-only route. An entry in parsers/aliases.toml that is not disabled and does not run poka-hex: no crate of its own, another module's wasm under a new id, name and extension.

  • Shows: A host routes the extension to a module that is a real parser of something.
  • Does not show: That the module understands this specific format. The parser tutorial asks for a corpus sample to be run first; the scorecard cannot see whether it was.

T5 Hex stub (backlog). Any manifest, not disabled, that is listed in parsers/unsupported.toml, runs the retired poka-hex module, or names the retired poka-hex-viewer as its viewer. Both are gone, so this tier is a guard against hex shells coming back.

  • Shows: The extension is recognised and a file opens as a hex dump.
  • Does not show: Any parsing. This is the pre-support state: kpi-stats.py reports the unsupported.toml stubs as a backlog, not coverage, but counts own-crate hex shells as full parsers; the scorecard does not.

T6 Disabled. disabled: true in a crate's plugin.json, or a disabled_reason on an aliases.toml route (SPEC.md section 2).

  • Shows: Nothing: the host skips registration. Kept for the id and as a record.
  • Does not show: Whatever depth the source has, it is not reachable.

What no depth tier can see: whether a parser's output is correct, how much of the format it decodes, and whether a poka-hex-viewer shell hides behind another viewer's name. For how much of the format is decoded, section 8 gives the breadth tier of the formats reviewed so far; for the rest, read the crate's module documentation and its tests/samples.rs.

8. Breadth tiers

A depth tier (section 2) says what kind of parser reads a format. A breadth tier says how much of the format Poka reads. It is not derived from the manifests: a reviewer assigns it from the parser source, its tests, the corpus and the format's own documentation, and writes it into the format's entry in docs/data/format-metadata/. Only formats that have a format page can be assessed, so this section covers those and says nothing about the rest of the registry. Counts are measured from those files; what a tier means is the reviewer's judgement, and docs/explanation/breadth-tiers.md says how far it can be trusted.

379 of 379 documented formats are assessed (100.0%).

Breadth tier Formats Which Basis
B1 Complete 12 BND, DIR, DWL, DWL2, GIF, MDD, MTD, PGM, QOI, RAS, RDD, SOE measured
B2 Broad 92 3MF, ASD, AWS-CREDENTIALS, BASH_HISTORY, BMP, CAPNP, CIN, CLASS, CNT, COOKIES, CRL, CRONTAB, CSR, CTB, D5MESH, DAE, DDY, DEM, DPX, DTB, DWG, EDN, EIO, ERR, EVT, FBS, FOLDED, FSTAB, GH, GHX, GLB, GLTF, GROUP, GZ, HDR, HOSTS, HTPASSWD, IDD, IDF, INF, INP, IPYNB, JKS, JPEG, JPG, JWKS, JWT, KNOWN_HOSTS, LNK, MAGNET, NETRC, OSW, P7B, P7S, PASSWD, PATCH, PCX, PDF, PEM, PGPASS, PNG, PPM, PRJ, REG, REGO, RS, SERVICES, SHA256SUM, SHADOW, SMK, SSHCONFIG, STL, STRACE, SUDOERS, SVELTE, SVG, SYSLOG, TAB, TBD, TGA, THRIFT, TLE, TTF, VCF, VUE, WEA, WEBP, WG, XCU, XER, XML, XPM measured
B3 Core 79 3DM, APNG, ASC, AUTOMATICDESTINATIONS-MS, B18, BDL, BEAM, CALLGRIND, CGM, CPUPROFILE, DCK, DDS, DER, DOC, DOCX, DOT, DRS, DSBXML, DXBC, DXF, DYF, DYN, ELF, EXE, FBX, GEOJSON, GML, GPKG, GPX, HEAPSNAPSHOT, IES, IFC, IFCZIP, JSTACK, JT, KML, KMZ, LDT, LUAC, MACHO, MFT, MID, MMD, NBT, NMAP, OBJ, ODB, ODP, ODS, ODT, OSM, OTT, OVPN, P12, P8, PCAP, PCAPNG, PF, PKPASS, PPROF, PPT, PPTX, PSD, PYC, RAD, SHP, STA, STEP, TCX, THEME, THMX, TSM, VSDX, WASM, WPG, XBE, XLSX, YANG, YAR measured
B4 Summary 184 7Z, ACE, ADA, ADF, AFM, AGDA, AHK, APK, APPIMAGE, APPLESCRIPT, APPX, AR, ARC, ARJ, BDF, BLO, BLP, BMFONT, BOCHS, BOM, BZ3, CAB, CAT, CBL, CFB, CLM, CORE, CPIO, CR, CRON, CYPHER, D.TS, D2, D64, DATA, DEB, DHALL, DLL, DMP, DSK, DYLIB, EBNF, EDB, EGG, ELM, EOT, EVTX, EXT4, F90, FEN, FNL, FNT, FON, FONTCONFIG, GB, GBA, GBS, GEN, GPT, GR, GRIB, HBS, HPROF, IMD, IMG-CRIU, IPTABLES, ISO, ISPROJ, JAR, JFR, JSONNET, KDBX, KK, KO, LEAN, LHA, LL, LUKS, LZ, LZ4, LZH, LZO, M2TS, MAP, MAR, MBR, MBTILES, MD-ROM, MDS, MIR, MLIR, MOJO, MP3, MP4, MPQ, MSI, MST, MSU, MTR, NES, NFT, NIM, NRG, NSF, NU, O, OCT, OSC, PAK, PAR2, PAS, PBF, PCF, PDB-DEBUG, PERFETTO-TRACE, PFA, PFB, PFM, PGN, PLI, PONY, PPK, PRG, PROLOG, PUML, PURS, PYI, RAR, RAR5, RBS, RES, RKT, ROC, RPM, SAV, SDG, SDW, SFC, SFD, SGF, SHX, SID, SKP, SMC, SNAPPY, SNOOP, SPC, SPV, SQLITE, SQLITE3, T64, TAR, TFM, THUMBCACHE, TLA, TM2, TMY3, TRACE.JSON, TSCN, TTC, TZX, UCI, UUE, V, VALA, VBS, VDI, VHD, VHDX, VLANG, VMDK, VPK, WAT, WAV, WIM, WOFF2, XAR, XEX, Z01, Z64, ZIG, ZIP, ZOO, ZST measured
B5 Identification 12 BR, BSDIFF, DB, DGN, EKD, FLAC, LZMA, OGG, UMX, WEBM, Z, ZPAQ measured
Not assessed 0 none measured
Total 379 measured

By group: one row per file in docs/data/format-metadata/, the formats it documents counted by breadth tier. A group with few formats has few format pages, not necessarily shallow parsers.

Group B1 B2 B3 B4 B5 Not assessed Formats
archives 0 0 1 52 5 0 58
audio-video 0 0 1 1 3 0 5
cad-3d 0 6 7 1 1 0 15
cad-bim 2 6 5 0 0 0 13
climate-light 0 1 7 4 0 0 12
documents 0 2 2 0 0 0 4
energyplus 4 6 1 2 0 0 13
fonts 1 0 0 15 0 0 16
games 0 0 2 27 1 0 30
gis 0 4 5 4 0 0 13
images 4 13 4 0 0 0 21
media-data 0 2 2 5 1 0 10
misc 0 2 2 0 0 0 4
office 1 1 10 2 0 0 14
source-code 0 11 3 42 0 0 56
system 0 34 26 24 0 0 84
system-pl 0 4 1 5 1 0 11
Total 12 92 79 184 12 0 379

Breadth tiers by group, as a heatmap (the table above has the same figures)