build-number
duration-label: 17m 0s
started-label-colon: time-days-ago
Respect speller config, suggestion weights, and deduplicate properly
Three defects found while measuring the sme speller through a pipeline.
**The SpellerConfig mirror had drifted.** cgspell keeps its own copy of
SpellerConfig to generate the TypeScript type, and it was missing
search_budget, word_split_weight and ReweightingConfig's curve. A
pipeline setting them got no type for them and they were dropped, so
search_budget never arrived and the search fell back to its legacy
10,000,000-iteration cap -- seconds on a single word, in a pipeline that
runs interactively.
Worse, the mirror was snake_case while divvun_fst's real SpellerConfig
is rename_all = "kebab-case". Serde fills a default for a key it does
not recognise, so every hyphenated option a pipeline set was silently
ignored: n-best fell back to 10 where sme asks for 100, and max-weight
and all three reweight penalties never applied either. Only single-word
keys -- beam, recase -- ever took effect. Proven directly against the
shipped speller: {"n-best": 3} returns 3 suggestions, {"n_best": 3}
returns 10. The mirror is now kebab-case too, so its own round-trip
through TryFrom stops losing fields.
**Suggestion order ignored the weights.** cgspell writes each
suggestion's total weight as <W:...>, and a CG rule can rewrite that
tag, but suggest never read it -- order was whatever order the readings
happened to be in. Today that is already weight order, since cgspell
emits in the speller's order and rayon's collect preserves it, so this
changes nothing on its own. What it changes is what a grammar can do:
CG3 has no reorder operation, so rewriting a weight is the only way a
rule could move a suggestion rather than drop it. Entries with no weight
are left where they are -- &SUGGEST forms come from the generator and
&SUGGESTWF ones are literal word-forms, neither on the speller's scale.
**Deduplication only removed adjacent repeats.** Vec::dedup is
consecutive-only, and sforms are gathered from several analysis groups,
so syncretism routinely puts one form in twice with another between
them: hálidit analyses as Inf, Prs Pl1, Prs Pl3 and Prt Sg2 of one
lemma, two of which generate háliidit, and the user was shown it twice
with háliidat in between. 51 of 282 measured suggestion lists carried a
duplicate, each one wasting a slot in the n-best the user sees. All
three call sites now keep the first occurrence and drop the rest.
7 new tests; suite 86 passing.
jobs-heading
divvun-actions cidivvun-actions ci
ran-in▶
build-aarch64-pc-windows-msvcuv venv --python 3.12; uv pip install pyyaml setuptools; $en...
ran-in▶
lib-build-x86_64-unknown-linux-gnu./x build-lib --target x86_64-unknown-linux-gnu
mkdir -p lib...
ran-in▶
lib-build-aarch64-apple-darwin./x build-lib --target aarch64-apple-darwin
mkdir -p libdivv...
ran-in▶
build-x86_64-unknown-linux-gnu./x build --target x86_64-unknown-linux-gnu
mv target/x86_64...
ran-in▶
aarch64-apple-darwin - Build./x build --target aarch64-apple-darwin
mv target/aarch64-ap...
ran-in▶
build-x86_64-pc-windows-msvcuv venv --python 3.12; uv pip install pyyaml setuptools; $en...
ran-in▶
lib-build-x86_64-pc-windows-msvcuv venv --python 3.12; uv pip install pyyaml setuptools; $en...
ran-in▶
lib-build-aarch64-pc-windows-msvcuv venv --python 3.12; uv pip install pyyaml setuptools; $en...
ran-in▶
aarch64-apple-darwin - Signecho '--- Downloading unsigned binary'
buildkite-agent artif...
not-started▶