open https://gitlab.synchro.net/main/sbbs/-/work_items/1247

## Problem

`?view=` on the web file index and the `list` path of `exec/archive.js` both call `Archive(filename).list()` on every request. Nothing is ever retained, so the same archive is opened and enumerated once per view, forever.

That has three costs:

1. **Repeated work.** Measured on Vertrauen over six days, 97,280 `?view=`
requests covered 70,199 distinct URLs, so 72% were the first-ever request
for that URL. A conventional expiring cache would ride at about a 28% hit
rate; the population is a breadth-first crawl of a large corpus, not a
working set.
2. **A crash vector.** Building the entry list in mozjs185 is what
`d777bda5c5` (patch-1-sell, 2026-06-01) deflects crawlers away from. The
deflection reduces exposure but the underlying extraction still happens on
every legitimate view.
3. **Guaranteed-useless work.** 178 of 600 sampled archive-extension files
fail to list at all, and 171 of those are `.arc`, which libarchive does not
support in any form. Every view of those re-attempts a doomed extraction and
returns an error page.

## Proposal

Persist the outcome of archive-content extraction in a per-area file, and have viewers read it instead of extracting.

### Storage

One file per file area, alongside the base it derives from:

```
data/dirs/<code>.contents
```

`data/dirs/` already holds multiple per-area extensions (`shd` `sdt` `sid` for the base, `sha` `sda` for header and data allocation, `hash`, `ini`, plus the legacy `ixb` `dat` `dab` `exb`), so this is consistent with existing practice. `contents` collides with none of them.

Rejected alternatives: one file per archive costs about 576 MB of 4 KB block waste to hold roughly 82 MB of content, plus 144k inodes, for areas that hold 27 records at the median. A single global file puts every writer on one hot file with no lifecycle tie to anything.

### Record shape

Keyed by filename within the area. Each record carries:

- the extracted listing, as compact entries of name, size, time
- `type`, so the same file can later hold image, audio or video metadata
without a format break
- the validity key: the file's size and mtime at extraction time
- a format version, and an **extractor identity/version**
- for failures, the outcome and reason instead of a listing

Two versions rather than one, because they protect different things. The format version protects the encoding. The extractor version protects the *verdict*: a negative record written today because libarchive cannot read ARC must be re-examined if ARC support is ever added, and nothing else would trigger that, since the file itself never changed.

### No size cap

Measured across 500 archives sampled at most 4 per area: mean listing 712 bytes, median 266 bytes and 9 entries, p99 10 KB and 330 entries, 34.4 bytes per entry. With 83.3% of the 144,259 file records being archives, the whole base projects to **about 82 MB uncapped**. A cap saves nothing worth having.

Note this is deliberately not a guard against the mozjs185 crash vector. That cost is paid while building the list in memory, before any of it reaches storage, so a stored-size limit cannot prevent it. If a guard is wanted it belongs in the extraction path and is a separate concern. The store helps with the crash vector in a different way: each pathological archive is extracted once ever rather than once per view.

### Read path

Three tiers, in this order:

1. Store hit and current (size and mtime match): serve it.
2. Miss or stale: extract, serve, and store the result.
3. Cannot extract: report unsupported, and store *that*, so the next view does
not retry.

Tier 3 matters more than it looks. It is what stops the 171-in-600 `.arc`
files from re-attempting a doomed extraction on every single view.

### Shared reader

`exec/archive.js` (`list` and `json`) and `exec/webfileindex.ssjs` (`view_archive()`) need byte-identical lookup, staleness comparison and fallback. That belongs in one `load()`-able library under `exec/load/`, with both callers thin. Two implementations of a staleness rule will drift, and the drift is invisible until someone is looking at a listing that does not match the file.

This is an added fast path, not a rewrite. `archive.js` also has `create`, `extract`, `read`, `type` and `install`, which still need real extraction and are untouched. Only `list` is wired into configuration
(`[viewer:17] cmd=?archive list %f`).

### archive.js takes a path, not an area

`archive.js` accepts a path or filename and knows nothing about area codes, and its configured invocation passes only `%f`. Passing the code instead would require a new `cmdstr()` specifier, and there is none for it today (checked all of A-Z plus the symbol specifiers in `xtrn.cpp`), so that would turn a pure-JS change into a C++ one.

Resolve it in JS instead, by reverse-mapping the path to an area:

1. Normalize the given path to absolute, and split into directory plus
filename.
2. Compare the directory against `file_area.dir[*].path`, respecting platform
case rules and the trailing separator that `dir.path` already carries.
3. On a match, look up the filename in that area's `.contents`.
4. On no match, there is no store. Extract live and do not persist, which is
correct: a file outside every file area has nowhere to persist to.

Cost is about 1,400 in-memory string comparisons per invocation against already-loaded configuration, memoizable per process. `archive.js` keeps its current interface exactly, and `[viewer:17]` needs no change.

Edge cases, all of which degrade to live extraction rather than misbehaving: paths reached via symlink or a different route will not match; two areas sharing a path simply get their own `.contents` with duplicate entries, which is harmless.

### Populator

`exec/inventory_archives.js` is rewritten to target the store instead of `auxdata`. The area iteration and extraction logic survives; the write target and the skip logic change. Two existing bugs disappear with the rewrite rather than needing fixes:

- It passes `readd_always: true` to `FileBase.update()`, which zeroes
`when_imported` (`js_filebase.cpp`), and `smb_addmsg()` re-stamps a zeroed
value to the current time (`smbadd.c:247`). A full run against every area
would mark the entire file base as newly imported.
- It skips any file that already has data rather than checking whether that
data is still valid, so it has no staleness handling at all.

With the three-tier read path in place, running the populator is a tuning decision (pre-empting the crawler, warming cold areas) rather than a prerequisite for correctness.

### ARC

`.arc` was 28.5% of sampled archive-extension files and is entirely unviewable today. Two extractors handle it, both already present on Vertrauen:

| tool | listing | notes |
|---|---|---|
| `lsar` / `unar` | `lsar -j` emits JSON | preserves filename case |
| `nomarch` | `nomarch -l` | lowercases filenames |

Both listed **80 of 80** sampled `.arc` files that libarchive rejects. `lsar -j`
is preferable: it needs no output parsing and returns `XADFileName`, `XADFileSize` and `XADLastModificationDate`, which is exactly what the viewers render.

The store makes this a populator concern rather than a viewer concern. Once a listing is stored, no viewer cares which tool produced it, and no per-format `[viewer:N]` entry is needed for ARC or anything else. Format knowledge collapses into one place.

## No C++ changes required

Everything needed exists in the JS API already: `File`, `file_size()`, `file_date()`, `system.data_dir`, `FileBase`, `Archive().list()`, and `system.popen()` for shelling out to `lsar`.

One caveat on `system.popen()`: its own documentation says "only functional on UNIX systems". It compiles everywhere, since `genwrap.h:315` maps `popen` to `_popen` on Windows, but `_popen` requires a console and fails inside a Windows service. `js_popen()` returns early on a NULL `FILE*` without setting a return value, so the caller sees `undefined`. The fallback must read that as "no capability" and drop to tier 3, rather than treating it as an empty listing and poisoning the store with bogus empty records. Net effect on Windows: ARC stays unsupported exactly as it is today, with no regression.

Making `Archive()` itself fall back to an external extractor would fix every consumer at once but is not proposed: it puts a process spawn inside a core object and inherits the same Windows limitation.

## Prior art, and why this differs

Archive contents were stored in the SMB file "tail" in `c5c17a98ad` (memo-43-dive, 2021-05-02) and removed ten days later in `2c5b131848` (city-14-relative, 2021-05-12) for two stated reasons: it badly slowed `upgrade_to_v319`, and there was no consumer of the data. `6b13e59979` (sync-3-shark, 2021-05-12) recorded the intended direction as JS and JSON rather than C and .ini, and `2066c74aa1` (layout-11-geek, 2023-12-25) left `exec/inventory_archives.js` as a proof of concept.

Both reasons have changed. `?view=` arrived in `50d04b24aa` (trio-35-tall, 2025-01-22) and is now the consumer, under crawler load. And the cost that
sank it was a property of the storage location, not of the idea: writing `auxdata` takes the remove-and-re-add path rather than the in-place `smb_putfile()` (`js_filebase.cpp`), under `smb_locksmbhdr()` on the area's `.shd` with a 30-second retry timeout, inflating `.sdt` in the process. A separate file has none of those properties, and the listing is derived data that is reconstructible at any time, so it does not belong in the authoritative record regardless.

A survey found the slate is clean: across all 1,404 configured areas, exactly one area (`fidonet_mbse_bbs`) holds any stored contents, 2 records, in the 2021 .ini format. There is nothing to migrate.

## Open questions

- Whether `?view=` should remain crawlable at all. The design does not depend
on the answer, but it decides whether the crawler warms the store as a side
effect or whether warming is the populator's job alone.
- Whether tier 2 should be allowed on the web path. Permitting it lets crawler
traffic populate the store for free; forbidding it keeps process spawning off
anonymous requests entirely. Given `?view=` traffic is almost entirely
crawler, this is a real trade rather than an obvious call.
- Orphan sweeping. 1,794 `.sid` files exist against 1,404 configured areas, so
about 390 orphaned bases are already sitting in `data/dirs/`. The store will
accumulate the same way, which argues for folding it into whatever ought to
clean those up rather than inventing a separate mechanism.

-- *Authored by Claude (Claude Code), on behalf of @rswindell*
--- SBBSecho 3.37-Linux
* Origin: Vertrauen - [vert/cvs/bbs].synchro.net (1:103/705)