Pilot vitest+v8 code coverage on ten packages

[tylerbutler]

Description

Pilots vitest with its v8 coverage provider as an alternative to the current c8-based test:coverage script on 10 packages across the runtime, loader, DDS, and common utility layers.

Motivation. Repo-wide code coverage was disabled in CI in April 2026 (tools/pipelines/build-client.yml, testCoverage: false) because c8 performance regressed on Node 22 and further on Node 24, causing test timeouts. This pilot introduces a faster alternative without touching the existing c8 path or CI config.

What this PR does

• Adds test:coverage:vitest to each of 10 packages.
• Adds vitest.config.ts per package, with package-specific excludes for fuzz/benchmark/describe-level-this.timeout tests that don't work under strict-ESM vitest.
• Adds common/build/build-common/vitest-test-setup.mjs — a pre-compiled ESM shim shared by all pilot packages. It aliases mocha's before/after to vitest's beforeAll/afterAll, strips the optional leading-string arg from beforeEach/afterEach (mocha allows beforeEach("name", fn), vitest doesn't), no-ops this.timeout()/this.retries()/this.skip() on the test context, and stubs getTestLogger (normally set by @fluid-internal/mocha-test-setup's beforeAll).
• Adds scripts/bench-coverage.sh + scripts/bench-coverage-all.sh — a hyperfine-based harness for reproducible c8-vs-vitest timing comparisons, per-package or across the pilot set.

Packages

Layer	Package	Runs against
DDS	@fluidframework/tree	lib/** (decorators)
DDS	@fluidframework/map	src/**
DDS	@fluidframework/matrix	src/**
DDS	@fluidframework/merge-tree	src/**
Runtime	@fluidframework/container-runtime	src/**
Runtime	@fluidframework/runtime-utils	src/**
Runtime	@fluidframework/id-compressor	src/**
Loader	@fluidframework/container-loader	src/**
Common	@fluidframework/core-utils	src/**
Common	@fluid-internal/client-utils	src/**

Why one package runs against lib/** and the rest against src/**

Vitest runs directly on source TypeScript via its esbuild transform. It's the idiomatic approach and means no build step is required before running coverage. 9 of the 10 packages use this.

@fluidframework/tree is the exception. It uses the standard-ES decorator syntax (@breakingClass, @breakingMethod in src/util/breakable.ts). OXC — vitest 4's TS transformer — doesn't currently lower these to runtime __decorate calls, so Node's VM can't execute the transformed source. tsc-compiled lib/** has already done the lowering, so pointing vitest there sidesteps the issue. The v8 coverage provider follows source maps back to src/** for line-accurate reporting. If OXC gains decorator-lowering support, the tree config can flip.

Timing (hyperfine, N=3 runs + 1 warmup, macOS on Node 24)

Measured via scripts/bench-coverage-all.sh. Full per-package .md + .json outputs land in .benchmarks/ locally (gitignored).

Package	c8+mocha (s)	vitest+v8 (s)	Speedup
core-utils	1.66 ± 0.02	0.99 ± 0.02	1.7×
client-utils	5.09 ± 0.24	0.99 ± 0.01	5.1×
runtime-utils	1.71 ± 0.02	1.27 ± 0.02	1.3×
id-compressor	10.06 ± 0.19	1.07 ± 0.02	9.4×
container-loader	2.22 ± 0.07	1.84 ± 0.04	1.2×
container-runtime	9.75 ± 0.18	4.11 ± 0.21	2.4×
map	8.09 ± 0.03	1.61 ± 0.01	5.0×
matrix	401.10 ± 29.36	13.62 ± 0.07	29.4×
merge-tree	775.64 ± 12.88	10.83 ± 0.16	71.6×
tree	151.35 ± 4.43	25.84 ± 0.21	5.9×
Sum of means	1366.67	62.16	22.0×

merge-tree and matrix are the dramatic cases — their c8+mocha paths spend 13 min and 6.7 min per run respectively; vitest+v8 brings them under 15 seconds. Smaller utility packages see modest wins because their tests are already fast; gains scale with test-suite size.

Coverage

Package	Tests	Pass %	Cov (stmt)
map	157/157	100%	92.75%
matrix	274/274	100%	90.23%
merge-tree	893/895 (+2s)	99.8%	84.60%
container-runtime	902/904	99.8%	77.84%
container-loader	224/224	100%	66.84%
runtime-utils	92/93	98.9%	56.41%
client-utils	30/30	100%	50.00%
core-utils	73/73	100%	33.51%
id-compressor	24/24	100%	31.03%
tree	7190/7770	92.5%	91.58%

Coverage caveats

Coverage numbers are not strictly apples-to-apples with the c8 baseline:

• Vitest excludes fuzz/perf/benchmark/this.timeout-at-describe-scope test files that it can't execute; their source contributions are missing from the numerator.
• For packages running against src/**, coverage reflects what the tests actually exercise — not what merely gets imported by the tsc-compiled output. For some packages (core-utils notably) this drops the % noticeably vs. the lib/** strategy, but the denominator is unchanged.
• client-utils coverage is limited to the mocha suite; the jest suite is out of scope for this pilot.

Known vitest-only test failures

Documented, not fixed here. None are vitest-compat issues — they're real assertions that behave differently under a single-ESM-pass vitest vs. the multi-pass mocha run:

• tree — 266 failures in codec/snapshot round-trip suites (likely filesystem/CWD-sensitive).
• container-runtime — 2 assertion edge cases.
• runtime-utils — 1 test has a nested it() inside another it(); mocha tolerates it, vitest rejects (this is genuinely a bug in the test).
• merge-tree — 2 skipped (existing it.skip in the source, not a vitest issue).

Reproducing the timings

# Per package:
scripts/bench-coverage.sh @fluidframework/map

# All 10:
scripts/bench-coverage-all.sh

# Env knobs:
BENCH_MODE=vitest scripts/bench-coverage.sh @fluidframework/tree 5 2

Requires hyperfine (brew install hyperfine). Outputs: .benchmarks/bench-<slug>.{md,json}.

Reviewer Guidance

The review process is outlined on this wiki page.

Draft until we've decided:

1. Is common/build/build-common the right home for vitest-test-setup.mjs? The file is pre-compiled ESM (no build step required), consistent with build-common's role as the canonical place for shared build/test configs (tsconfig.test.node16.json, api-extractor-base.json, etc.). Alternative would be a new @fluid-internal/vitest-test-setup package paralleling @fluid-internal/mocha-test-setup.
2. Is 10 packages the right pilot scope, or should we go further (or narrower) before committing to the approach?
3. Not touching CI. Reenabling coverage in CI is intentionally out of scope — we can do that in a follow-up once the approach is agreed.

How to try it

# src/** packages — no build step needed
pnpm --filter @fluidframework/map run test:coverage:vitest
open packages/dds/map/nyc/report-vitest/index.html

# tree still needs a build
pnpm --filter @fluidframework/tree run build
pnpm --filter @fluidframework/tree run test:coverage:vitest
open packages/dds/tree/nyc/report-vitest/index.html

[tylerbutler]

/azp run Build - protocol-definitions,Build - test-tools,server-gitrest,server-gitssh,server-routerlicious,Build - client packages,repo-policy-check,Build - build-tools

[azure-pipelines]

Azure Pipelines successfully started running 5 pipeline(s).