divvun-runtime

Ensure pre-blank on ADDED words

Ensure pre-blank on ADDED words

Add COPYCOHORT to CG tag list so we drop it when input has cg trace

divvunspell 6f17d4d: suggestions keep the lexicon's capitals Brings the speller's mixed-case fixes to the grammar checkers: iPod and NSR:a are no longer recased into IPod and NSR:A, and the case and separator probes find DNA-prosessajda, IKT-vædtsaga or Gasska-Nordlánda from lower-case input.

Speller config: carry initial-case-weight; divvunspell 7a0cada divvunspell's new initial-case-weight lets a correction start with a capital the error model cannot type. The mirror declares it, so a pipeline or a bundled speller config that sets it keeps it. divvun-fst moves from b786aacf to 7a0cada, which brings the key and the search speed-ups since, and drops node-pool-size from the speller's config. The mirror drops it too.

--version gives the build date and git revision divvun-runtime --version printed only the package version (0.4.0); the build date and commit were only in -VV's debug dump. It now prints "Divvun Runtime v0.4.0 (2026-10-07, e59bb6e)", with -dirty for a modified tree, the same shape as Divvun HFST and divvunspell; -VV adds the full build information after that line.

Keep a regional message file's whole locale, and declare every FFI export extract_language_code split errors-<lang>.ftl on the last dash, so errors-se-NO.ftl registered its messages under "NO" and nothing could request them. Take everything after the first dash instead. DRT_PipelineHandle_cancel and DRT_Bundle_runPipeline were exported but missing from the C header; declare them. Cancel was also missing from link_keep() and the CLI's exported symbols.

Let callers ask the speller whether a string is an entry When a word is misspelled, LibreOffice re-checks it joined with a neighbour ("okta okta") to find multi-word dictionary entries, and drops the underline if that comes back valid. Running the pipeline cannot answer that: it tokenises the string and only reports that each token is spelled correctly. So the extension rejects any input with whitespace in it, which underlines the first word of real multi-word entries (divvunspell-libreoffice#13). The cgspell acceptor can answer it directly. Add Cgspell::is_correct, which looks the string up with is_correct_with_config under cgspell's own config, so recasing behaves as it does in the pipeline, and expose it as DRT_Bundle_isCorrect. cffi's BoolMarshaler has no fallible form, so a small BoolResultMarshaler reports errors through the callback and returns false. A character outside the acceptor's alphabet maps to its unknown symbol, or to epsilon when it has none, so a space could match where the acceptor has no space at all. Refuse strings with a space outright when " " is not in the alphabet, and refuse any other whitespace always: LibreOffice joins with a plain space. Against se.drb, whose acceptor has " " as a symbol: "sápmelaš", "dan dihte", "juohke sajis", "New York" and "Sri Lanka" are entries; "sápmelas", "okta okta", "Mun Mun", "dihte dan", "dan dihte" and "dan dihte" with a no-break space are not. The cgspell_is_correct example runs the same lookup from the command line.

Record the locales a bundle serves as drb.locales A consumer that wants to offer regional variants of a language -- se-NO vs se-SE, sjd-RU -- has had to carry its own language table to know which regions exist. The LibreOffice extension ships a 332-entry locales.json scraped from LibreOffice's isolang.cxx in 2022 for exactly this, and it goes stale the moment a language is added. The language repos already declare this, in manifest.toml's windows.extra_locales: { se-NO = "se", se-SE = "se", se-FI = "se" } for sme, { sjd-RU = "sjd" } for sjd. Carry it into the bundle so the fact travels with the artifact instead of being duplicated per consumer. bundle --locales "se-NO,se-SE,se-FI" -> drb.locales alongside the existing drb.type/name/version, and shown by `list`. Reading it needs a C-ABI accessor, and the two added here are general rather than one-per-attribute: DRT_Bundle_metadataAttr and DRT_Bundle_metadataKeys. Both take a path, not a bundle handle, and read the box trailer only -- no pipeline parse, no asset access -- because the caller wants to scan every installed bundle at startup and some are over a gigabyte. An absent key returns an empty slice and does not raise, so bundles built before this attribute existed read as absent rather than failing; consumers keep their fallback until bundles are rebuilt. divvun-runtime deliberately does not learn to read manifest.toml: it is a generic tool and manifest.toml is a Divvun packaging convention, so the value arrives as a CLI argument exactly like type/name/version do. Verified end to end over the C ABI: " sjd-RU , , se-NO " normalises to "sjd-RU,se-NO", and the installed sjd.drb (built before this) reports keys ["drb.type","drb.name","drb.version"] with drb.locales absent.

Record the locales a bundle serves as drb.locales A consumer that wants to offer regional variants of a language -- se-NO vs se-SE, sjd-RU -- has had to carry its own language table to know which regions exist. The LibreOffice extension ships a 332-entry locales.json scraped from LibreOffice's isolang.cxx in 2022 for exactly this, and it goes stale the moment a language is added. The language repos already declare this, in manifest.toml's windows.extra_locales: { se-NO = "se", se-SE = "se", se-FI = "se" } for sme, { sjd-RU = "sjd" } for sjd. Carry it into the bundle so the fact travels with the artifact instead of being duplicated per consumer. bundle --locales "se-NO,se-SE,se-FI" -> drb.locales alongside the existing drb.type/name/version, and shown by `list`. Reading it needs a C-ABI accessor, and the two added here are general rather than one-per-attribute: DRT_Bundle_metadataAttr and DRT_Bundle_metadataKeys. Both take a path, not a bundle handle, and read the box trailer only -- no pipeline parse, no asset access -- because the caller wants to scan every installed bundle at startup and some are over a gigabyte. An absent key returns an empty slice and does not raise, so bundles built before this attribute existed read as absent rather than failing; consumers keep their fallback until bundles are rebuilt. divvun-runtime deliberately does not learn to read manifest.toml: it is a generic tool and manifest.toml is a Divvun packaging convention, so the value arrives as a CLI argument exactly like type/name/version do. Verified end to end over the C ABI: " sjd-RU , , se-NO " normalises to "sjd-RU,se-NO", and the installed sjd.drb (built before this) reports keys ["drb.type","drb.name","drb.version"] with drb.locales absent.

Stop Windows release builds running out of memory, and fix the cdylib link Two failures on x86_64-pc-windows-msvc. rustc-LLVM ERROR: out of memory Allocation failed [profile.release] sets lto = true, codegen-units = 1 and debug = "full". Across executorch, onig, zstd, lzma and blake3 that puts the entire program in a single LLVM module with full debug info and exhausts the agent. Override to thin LTO, debug = 1 and no incremental via CARGO_PROFILE_* in the Windows branch of getEnvVars, so only Windows changes and the macOS and Linux release artifacts keep the original settings. Incremental does nothing under LTO anyway. lld-link: error: subsystem must be defined lld-link infers the subsystem from the entry point, and with everything merged into one LTO object it stops finding _DllMainCRTStartup when linking divvun-runtime-ffi's cdylib. The override dated from the repo's initial import with no stated rationale; rustc defaults to MSVC link.exe for msvc targets, which infers this correctly. crt-static is retained on both targets. Neither change is verified on Windows -- there is no msvc toolchain here.

Stop Windows release builds running out of memory, and fix the cdylib link Two failures on x86_64-pc-windows-msvc. rustc-LLVM ERROR: out of memory Allocation failed [profile.release] sets lto = true, codegen-units = 1 and debug = "full". Across executorch, onig, zstd, lzma and blake3 that puts the entire program in a single LLVM module with full debug info and exhausts the agent. Override to thin LTO, debug = 1 and no incremental via CARGO_PROFILE_* in the Windows branch of getEnvVars, so only Windows changes and the macOS and Linux release artifacts keep the original settings. Incremental does nothing under LTO anyway. lld-link: error: subsystem must be defined lld-link infers the subsystem from the entry point, and with everything merged into one LTO object it stops finding _DllMainCRTStartup when linking divvun-runtime-ffi's cdylib. The override dated from the repo's initial import with no stated rationale; rustc defaults to MSVC link.exe for msvc targets, which infers this correctly. crt-static is retained on both targets. Neither change is verified on Windows -- there is no msvc toolchain here.

Build Windows ARM64 with cargo instead of cross needsCrossCompile had same-OS cases for apple and linux but not windows, so an x86_64 Windows host targeting aarch64-pc-windows-msvc fell through to the `host !== target` catch-all and picked cross, failing with "Failed to spawn 'cross': entity not found". Installing cross on the Windows agents would not have helped: it drives Linux Docker images and has no MSVC support at all. Windows ARM64 cross-compiles natively via the ARM64 MSVC tools, which the agents already have. Generalise the per-OS cases into an osFamily helper (android checked first, since its triples also match linux) so same-OS/different-arch consistently means cargo, and replace the catch-all with explicit routing. Combinations with no working toolchain, such as an Apple target from a non-Apple host, now throw up front rather than dispatching to a tool that cannot service them and failing later with a confusing error.

Build Windows ARM64 with cargo instead of cross needsCrossCompile had same-OS cases for apple and linux but not windows, so an x86_64 Windows host targeting aarch64-pc-windows-msvc fell through to the `host !== target` catch-all and picked cross, failing with "Failed to spawn 'cross': entity not found". Installing cross on the Windows agents would not have helped: it drives Linux Docker images and has no MSVC support at all. Windows ARM64 cross-compiles natively via the ARM64 MSVC tools, which the agents already have. Generalise the per-OS cases into an osFamily helper (android checked first, since its triples also match linux) so same-OS/different-arch consistently means cargo, and replace the catch-all with explicit routing. Combinations with no working toolchain, such as an Apple target from a non-Apple host, now throw up front rather than dispatching to a tool that cannot service them and failing later with a confusing error.

Build Windows ARM64 with cargo instead of cross needsCrossCompile had same-OS cases for apple and linux but not windows, so an x86_64 Windows host targeting aarch64-pc-windows-msvc fell through to the `host !== target` catch-all and picked cross, failing with "Failed to spawn 'cross': entity not found". Installing cross on the Windows agents would not have helped: it drives Linux Docker images and has no MSVC support at all. Windows ARM64 cross-compiles natively via the ARM64 MSVC tools, which the agents already have. Generalise the per-OS cases into an osFamily helper (android checked first, since its triples also match linux) so same-OS/different-arch consistently means cargo, and replace the catch-all with explicit routing. Combinations with no working toolchain, such as an Apple target from a non-Apple host, now throw up front rather than dispatching to a tool that cannot service them and failing later with a confusing error.

Build Windows ARM64 with cargo instead of cross needsCrossCompile had same-OS cases for apple and linux but not windows, so an x86_64 Windows host targeting aarch64-pc-windows-msvc fell through to the `host !== target` catch-all and picked cross, failing with "Failed to spawn 'cross': entity not found". Installing cross on the Windows agents would not have helped: it drives Linux Docker images and has no MSVC support at all. Windows ARM64 cross-compiles natively via the ARM64 MSVC tools, which the agents already have. Generalise the per-OS cases into an osFamily helper (android checked first, since its triples also match linux) so same-OS/different-arch consistently means cargo, and replace the catch-all with explicit routing. Combinations with no working toolchain, such as an Apple target from a non-Apple host, now throw up front rather than dispatching to a tool that cannot service them and failing later with a confusing error.

Build Windows ARM64 with cargo instead of cross needsCrossCompile had same-OS cases for apple and linux but not windows, so an x86_64 Windows host targeting aarch64-pc-windows-msvc fell through to the `host !== target` catch-all and picked cross, failing with "Failed to spawn 'cross': entity not found". Installing cross on the Windows agents would not have helped: it drives Linux Docker images and has no MSVC support at all. Windows ARM64 cross-compiles natively via the ARM64 MSVC tools, which the agents already have. Generalise the per-OS cases into an osFamily helper (android checked first, since its triples also match linux) so same-OS/different-arch consistently means cargo, and replace the catch-all with explicit routing. Combinations with no working toolchain, such as an Apple target from a non-Apple host, now throw up front rather than dispatching to a tool that cannot service them and failing later with a confusing error.

Update Cargo.lock to current dependency versions Pulls divvun-fst (divvunspell) 6edc46e1 -> b786aacf and hfst-rs 0509ff05 -> 3ed8b683, plus ~80 crates.io bumps. serde_json and the other parsers are unchanged. Verified with ./x build-lib and by running the sjd and sma proofing bundles end to end through the LibreOffice extension's C ABI.

Drop fromKebabKeys; carry every speller config key through the mirror The helper existed so a pipeline could spread the wire-format speller config over these bindings' typed options, but it returned any, so it erased the config's type checking rather than bridging it — and being a generated export, it only existed in binaries newer than the builder running the job, which is how it broke the sme grammar checker. Gone; pipelines cast the imported JSON to the generated type instead, which type-checks their overrides for real. The mirror also declared continuation-marker where the speller has completion-marker, and lacked boundary-edit-weight entirely, so serde dropped both from every grammar checker: sme's measured boundary edit weight never reached the pipeline it was tuned for. The mirror now matches the speller's own config, and a test fails if a key it carries ever stops surviving the round trip.

Single-flight the asset caches The caches were lookup-then-intern: a miss loaded and the intern kept whichever copy won. Correct at rest, but a host growing a worker pool builds its handles concurrently, and every one of them missed before any had interned - measured on an sme bundle at eight workers, that is 24 CG-3 grammar loads for 3 grammars and 7 tokenizer core loads for 1 core, a transient resident peak of ~2.1 GiB for a process that settles at a third of that, and every one of those loads is ~0.5 s of CPU thrown away. The convergence also hid itself: the losers adopt the winner and drop their copies, so the steady state looks right while the peak is what OOMs a container. cache_get_or_load now owns the whole miss path: each key holds its own slot lock, the first caller loads under it, racers wait for the result, and a failed load releases the key so the next caller retries. Waiting blocks the thread, so the hfst tokenizer and CG-3 grammar sites run the lookup inside the spawn_blocking task they already load on; the lookup transducer and speller sites already loaded synchronously in their constructors and now wait there instead. cache_peek keeps the cheap fast path for the two sites that mmap before loading. Loads for different keys still run in parallel - the lock is per key, not per cache. 99 lib tests pass in debug and release; the new racing_callers test pins the property itself (8 racers, one load).