Build performance benchmarking suite and analyze computational scaling limits

[tinghf]

Build a repeatable benchmarking harness for MOSAIC calibration and use it to characterize computational scaling limits.
Reproduce and extend benchmark scenarios, including single-country runs, small multi-country systems, and full 40-location runs.
Automatically record runtime, memory usage, and I/O across problem sizes and hardware configurations.
Analyze how runtime and resources scale with number of locations and parameters.
Summarize safe operating ranges and recommend batching, staged estimation, or hierarchical strategies where appropriate.

[tinghf]

Benchmarking infrastructure: We built and validated a production-ready distributed computing platform for MOSAIC calibration using Docker containers on Azure with Dask/Coiled orchestration, supporting 100+ parallel workers. Single-country calibration (50K simulations) has been validated end-to-end with confirmed numerical equivalence to the local R pipeline. The systematic benchmarking campaign — sweeping across problem sizes (1 to 40 locations), recording structured performance metrics, and producing scaling analyses and operational recommendations — is queued as the next phase now that the infrastructure is stable.

[tinghf]

A brief breakdown in misc. sub-goals of this workitem:

Progress Assessment

What's Been Accomplished

The primary focus over the past few weeks has been standing up the distributed computing infrastructure required to run benchmarks at scale:

• Docker + Azure Container Registry: Production image (mosaic-worker:latest, 4.75 GB) with MOSAIC, laser-cholera, and all dependencies pre-installed. Workers start in ~30 seconds vs 15–30 minutes with the previous conda approach.
• Coiled/Dask integration: run_MOSAIC_dask() (2,028 lines) orchestrates calibration across 100+ cloud workers on Azure VMs. Resolved rate limiting, OOM, TLS disconnection, and threading safety issues across 15+ debug sessions.
• Equivalence validation: Confirmed Dask and local R pipelines produce identical parameter samples and expected simulation divergence (stochastic). Three rounds of validation with a structured comparison framework.
• Single-country calibration validated: ETH at 1K and 50K simulations, both fixed and adaptive modes, completed successfully.

Status Against Goals

Goal	Status	Notes
Repeatable benchmarking harness	🟡 Partial	Test scripts exist but are one-off configs, not a parameterized harness that sweeps problem sizes
Scenario coverage (1, multi, 40 locations)	🟡 Partial	Single-country (ETH) only. Multi-country and full 40-location runs not yet attempted
Automated metric recording (runtime, memory, I/O)	🟡 Partial	Ad hoc logging during runs; no structured output (CSV/parquet) for analysis
Scaling analysis across problem sizes	🔴 Not started	Only one location tested — no scaling curves or bottleneck identification
Safe operating ranges & strategy recommendations	🔴 Not started	Cost estimates exist (~$15/1K sims, ~$200/8 countries) but no empirical scaling data to base recommendations on

Next Steps

1. Build parameterized harness — accepts (n_locations, n_sims, n_workers), writes structured timing/memory results to parquet
2. Run scaling sweep — 1, 3, 8, 20, 40 locations across worker counts
3. Analyze scaling curves — identify where runtime/memory becomes limiting
4. Write recommendations — safe operating ranges, batching strategies, staged estimation guidance