divvun-runtime

Update divvunspell so pipeline speller configs take effect The lockfile still pointed at fc30fe67. Moving it to current main picks up the snake_case config aliases, without which every hyphenated speller option a pipeline sets is silently ignored -- the TypeScript binding is generated from Rust field names and so emits n_best, while the config deserializes n-best and fills a default for anything else. Measured on lang-sme's own pipeline.ts, unchanged, one word: 8 spelled readings before, 170 after. It asks for n_best 100 and had been getting the default 10, with max_weight and all three reweight penalties equally inert; only beam and recase, the keys without a hyphen, ever applied. The jump also brings in the divvunspell work done since fc30fe67 -- best-first search, search dedup, lazy subset construction, the case and boundary probes. Workspace tests pass, and the sme grammar checker and speller pipelines both still run.

Respect speller config, suggestion weights, and deduplicate properly Three defects found while measuring the sme speller through a pipeline. **The SpellerConfig mirror had drifted.** cgspell keeps its own copy of SpellerConfig to generate the TypeScript type, and it was missing search_budget, word_split_weight and ReweightingConfig's curve. A pipeline setting them got no type for them and they were dropped, so search_budget never arrived and the search fell back to its legacy 10,000,000-iteration cap -- seconds on a single word, in a pipeline that runs interactively. Worse, the mirror was snake_case while divvun_fst's real SpellerConfig is rename_all = "kebab-case". Serde fills a default for a key it does not recognise, so every hyphenated option a pipeline set was silently ignored: n-best fell back to 10 where sme asks for 100, and max-weight and all three reweight penalties never applied either. Only single-word keys -- beam, recase -- ever took effect. Proven directly against the shipped speller: {"n-best": 3} returns 3 suggestions, {"n_best": 3} returns 10. The mirror is now kebab-case too, so its own round-trip through TryFrom stops losing fields. **Suggestion order ignored the weights.** cgspell writes each suggestion's total weight as <W:...>, and a CG rule can rewrite that tag, but suggest never read it -- order was whatever order the readings happened to be in. Today that is already weight order, since cgspell emits in the speller's order and rayon's collect preserves it, so this changes nothing on its own. What it changes is what a grammar can do: CG3 has no reorder operation, so rewriting a weight is the only way a rule could move a suggestion rather than drop it. Entries with no weight are left where they are -- &SUGGEST forms come from the generator and &SUGGESTWF ones are literal word-forms, neither on the speller's scale. **Deduplication only removed adjacent repeats.** Vec::dedup is consecutive-only, and sforms are gathered from several analysis groups, so syncretism routinely puts one form in twice with another between them: hálidit analyses as Inf, Prs Pl1, Prs Pl3 and Prt Sg2 of one lemma, two of which generate háliidit, and the user was shown it twice with háliidat in between. 51 of 282 measured suggestion lists carried a duplicate, each one wasting a slot in the n-best the user sees. All three call sites now keep the first occurrence and drop the rest. 7 new tests; suite 86 passing.

Respect speller config, suggestion weights, and deduplicate properly Three defects found while measuring the sme speller through a pipeline. **The SpellerConfig mirror had drifted.** cgspell keeps its own copy of SpellerConfig to generate the TypeScript type, and it was missing search_budget, word_split_weight and ReweightingConfig's curve. A pipeline setting them got no type for them and they were dropped, so search_budget never arrived and the search fell back to its legacy 10,000,000-iteration cap -- seconds on a single word, in a pipeline that runs interactively. Worse, the mirror was snake_case while divvun_fst's real SpellerConfig is rename_all = "kebab-case". Serde fills a default for a key it does not recognise, so every hyphenated option a pipeline set was silently ignored: n-best fell back to 10 where sme asks for 100, and max-weight and all three reweight penalties never applied either. Only single-word keys -- beam, recase -- ever took effect. Proven directly against the shipped speller: {"n-best": 3} returns 3 suggestions, {"n_best": 3} returns 10. The mirror is now kebab-case too, so its own round-trip through TryFrom stops losing fields. **Suggestion order ignored the weights.** cgspell writes each suggestion's total weight as <W:...>, and a CG rule can rewrite that tag, but suggest never read it -- order was whatever order the readings happened to be in. Today that is already weight order, since cgspell emits in the speller's order and rayon's collect preserves it, so this changes nothing on its own. What it changes is what a grammar can do: CG3 has no reorder operation, so rewriting a weight is the only way a rule could move a suggestion rather than drop it. Entries with no weight are left where they are -- &SUGGEST forms come from the generator and &SUGGESTWF ones are literal word-forms, neither on the speller's scale. **Deduplication only removed adjacent repeats.** Vec::dedup is consecutive-only, and sforms are gathered from several analysis groups, so syncretism routinely puts one form in twice with another between them: hálidit analyses as Inf, Prs Pl1, Prs Pl3 and Prt Sg2 of one lemma, two of which generate háliidit, and the user was shown it twice with háliidat in between. 51 of 282 measured suggestion lists carried a duplicate, each one wasting a slot in the n-best the user sees. All three call sites now keep the first occurrence and drop the rest. 7 new tests; suite 86 passing.

suggest: look up error messages by raw tag before the encoded form Every error tag was run through encode_unicode_identifier before being handed to Fluent, but the .ftl files declare identifiers literally, so any tag containing an uppercase or non-ASCII character could never match: msyn-Sg3-ConNeg was looked up as msyn-sg3-conneg, and msyn-gen-numeral-jahkasaš as msyn-gen-numeral-jahkasa_u0161. A miss degrades silently to the raw tag, so users saw the bare error code where the Sámi feedback title should have been — in the sidebar and in the error-type list built by error_preferences. The encoding became redundant when the forked parser gained Unicode-identifier support (divvun/fluent-rs 727b532), leaving the lookup addressing keys that nothing writes. Try the tag verbatim first, then fall back to the encoded form, so .ftl files following either convention resolve. Resolving the 199 tags in lang-sme's errors.json against errors-se.ftl and errors-en.ftl goes from 82 se / 22 en / 95 raw to 132 / 53 / 14. The first column matches what api.giellalt.org serves today, which is what identified the encoder as the cause; the remaining 67 are absent from errors-se.ftl rather than misaddressed, and need feedback text written. 79 tests pass.

cgspell: analyse suggestions with the lexicon only analyze_output runs lexicon∘errmodel, so each suggestion was decorated with the analyses of its spelling neighbours (a suggestion one cheap edit from an infinitive claimed the infinitive reading). Upstream libdivvun uses lexicon-only analysis (speller->analyseSymbols, mode Lookup); this port regression made morphology-keyed suggestion filtering in CG impossible in principle — a wrong inflection whose neighbour licenses the morphology could never be removed. Now analyze_input, with a de-capitalisation fallback: the acceptor holds lower-case forms, so recased suggestions (sentence-initial Juohkehaš) would otherwise lose their readings and vanish. Found while prototyping context-sensitive speller-suggestion filtering: with correct analyses, 9 CG REMOVE rules gain +17 top-1 / 0 harmed on a 3,992-sentence error corpus; with the bug, the same rules move nothing. 79 tests pass.

Bump box for real

Bump box

bump vers

Fold blanktag's three blank-emitting loops into one The filter-out-BOS/EOS-then-emit sequence was written out three times, so teaching it about StreamCmd meant editing all three — and each ended in a `_ => {}` that would have silently dropped anything they missed. emit_blanks matches Block exhaustively instead, so the next variant is one compile error rather than three dropped blocks. The marker strings were also spelled out in all three filters on top of the two consts; they are now consts used as patterns.

Parse traced readings and stream commands instead of calling them text Output::lines was a three-way branch with a catch-all, while CG_LINE 850 lines below it encodes five alternatives. Two of them fell through to Block::Text: `;\t+…` traced removed readings (group 8) and <STREAMCMD:…> (group 7). blanktag marks Text as not part of the stream by prefixing ';', which is correct for foreign lines and destroys these two — a trace line came back out as `;;\t"mun" …` and a flush as `;<STREAMCMD:FLUSH>`, commented out. Both also got folded into blanktag's whitespace FST lookup, where the C++ only ever puts `:` blanks. Removed readings go *into* the cohort as Reading.removed rather than beside it, matching the C++ (which collects them in `readings`). As a sibling block they would have been deferred past their own cohort: iter only drains the pending-text deque when no cohort is open. The ';' is stripped in lines(), before rfind('\t'), so a removed sub-reading's depth is right rather than off by one. Display re-emits it, so a trace stream round-trips byte-for-byte. Cohort::kept() is the analysis view. It is not, however, usable everywhere: three clusters index-couple, and filtering one end of one desyncs the indices and panics when a cohort's last reading is removed. So indices stay over the whole vector and the filtering happens wherever an index space is owned — build_reading_hierarchy emits no node for a removed reading, keeping speech's three indexing sites valid untouched; group_readings keeps removed slots so run_cg re-emits verbatim for free, and generate_group plus the two suggest checks filter instead. proc_reading's fold now walks a kept index list, counting the '#' separator off kept rather than total. Two latent bugs fell out. cgspell's re-emit and blanktag's process_cohort both rebuilt reading lines by hand without the ';', so a removed reading would have been silently promoted to a kept one — and blanktag would have appended whitespace tags to it, which blanktag.cpp:90 explicitly refuses to do. Both go through Display now. cgspell's is_unknown no longer fires on a '?' the grammar already removed and hands the cohort to the speller.

Stop parsing a blank line as an empty text block Output::lines classified every line not starting with `"` or a tab as Line::Text, so a blank line came through as Block::Text("") — a null wearing the name of a pass-through line. blanktag marks Text blocks as not part of the stream by prefixing them with ';', so a stream that ended in a blank line grew a bare ';' on the end of gramcheck output, looking for all the world like trace leaking in. A blank line carries nothing in a CG stream; inter-cohort whitespace arrives as a ':'-prefixed superblank. So drop it in lines() rather than invent a block for it — the for/return skeleton there was already shaped for a filter it never performed. blanktag's ';' semantics are untouched.

Clear the remaining compiler warnings suggest: squiggle_bounds never used its fallback cohort (neither does the C++ it was ported from), the final-cohort path advanced a `pos` nothing reads after it, `reading_lines` was never written, and `trace_removed_readings` was never written or read — run_cg re-emits stream blocks verbatim instead of rebuilding them from Cohort, so the trace lines C++ collected per cohort come through as text blocks. ts: the schema-aware type string was computed and dropped; `returns` is a runtime type tag and the schema travels in its own field. ast: the streaming path's multiple-input arm is still a todo!(), so say what is missing when it fires. playground: get_active_tab/_mut had no callers; tabs are found by id.

Update hfst to 90531dff Nothing in the modules needed changing; foma did. The new hfst calls foma::constructions::fsm_substitute_pair, which 0.4.2 does not have, and `cargo update -p hfst` leaves registry deps alone — so foma goes to 0.4.4 with it, and nfst-* to 0.3.0.

Don't terminate run output that already ends in a newline A CG3 stream ends in a newline of its own, so writing one after it put a blank line between the output and the prompt. Only terminate a string value that doesn't already end in one; JSON and the other kinds still get their newline.

Update divvun-fst to fc30fe6 Reweighting no longer overrules the error model, so the nuvviDspeller version strings come back at their authored weights and in authored order. Pulls box-format 859053f7 along with it, since divvunspell requires it.

Update cg3 to f6bc102b Recoverable grammar-load failures no longer print a Rust panic through the default hook before the error is returned.

Take CG-3 grammar errors from the engine cg3-rs now compiles tag patterns through a single ICU-compatible seam and returns the diagnostics — tag text, pattern, cause, line — for every bad tag in one load, so the parse entry points are `Result<(), Cg3Error>` rather than a count in the Ok arm. That removes the reason this module installed a thread-local tracing subscriber around the parse to scrape the engine's log output, and the reason it string-matched `uregex_open returned ... trying to parse tag` back apart. Format the structured error instead. The `\Q...\E` tag in the Northern Sami disambiguator compiles now, so that pipeline loads.

Take Fluent error positions from the parser divvun/fluent-rs now reports the character that actually broke an entry rather than the indentation of the line above it, so the loader no longer has to re-scan the discarded slice guessing which of three known mistakes it was. What's left is phrasing: the parser's wording assumes you know the Fluent grammar, and these files are generated, so the same few mistakes recur and are worth naming. The missing-`=` case now points where the `=` should have been rather than at the attribute it belongs to.

Point Fluent parse errors at the actual mistake fluent-syntax reports where it gave up — the top of the entry it had to discard — not where the mistake is, so every error in errors-se.ftl came out as `662:1: Expected one of "a-zA-Z, Unicode letters, hyphen, or underscore"` with a caret under the indentation. Look inside the discarded slice for what actually breaks these files: a placeable that isn't one (`{€1}` where `{$1}` was meant), an attribute missing its `=`, a line indented with a tab. Report the whole file as one block, each error with its own line, a caret under the span, and what to do about it; fall back to the parser's own wording when nothing is recognised.

Fix #50: keep multiword lemmas intact in cgspell readings The lemma was taken as the first whitespace-separated chunk of the analysis, which chopped multiword entries ("Divvun speller for Northern Sami") down to their first word. Split on the acceptor's own symbol table instead: tags are its multi-character symbols, so the lemma ends where the rest of the analysis parses entirely as tags.