Skip to content

CLI-TEST-REPORT.md

Latest commit

 

History

History
168 lines (128 loc) · 7.61 KB

CLI-TEST-REPORT.md

File metadata and controls

168 lines (128 loc) · 7.61 KB

DevOps-OS CLI — Test Report (v0.2.0)

Run date: 2026-03-08
CLI version: 0.2.0
Python: 3.12.3
Runner: python -m pytest cli/test_cli.py -v

Summary

Metric Value
Total tests 52
✅ Passed 52
❌ Failed 0
⏭️ Skipped 0
⚠️ xfailed 0
Duration ~10 s

Result: ALL TESTS PASS ✅

CLI Commands Tested

Top-level CLI

Command Test Result
devopsos --help test_help ✅
devopsos --version test_version_flag_long ✅
devopsos -V test_version_flag_short ✅
devopsos --version output matches cli.__version__ test_version_matches_package ✅

devopsos init

Command / Behaviour Test Result
init --help shows --dir option test_init_help_shows_dir_option ✅
init --dir <path> creates .devcontainer/ in target dir test_init_dir_option_creates_devcontainer_in_specified_dir ✅
Checkbox prompt includes Space-to-toggle instruction test_init_checkbox_includes_space_instruction ✅
Selected tools are written to devcontainer.json test_init_selections_written_to_config ✅

devopsos scaffold

Command / Behaviour Test Result
scaffold --help lists all 7 subcommands test_scaffold_help_lists_new_targets · test_scaffold_help_shows_all_subcommands ✅
scaffold gha --help shows all GHA-specific options test_scaffold_target_help_shows_native_options ✅
Unknown target exits with a clean error test_scaffold_unknown ✅
Each subcommand with no options prints help + exits 0 test_scaffold_no_opts_shows_help ✅

scaffold gha — GitHub Actions

Command Test Result
scaffold gha --type build --name my-app generates .github/workflows/*.yml test_scaffold_gha_via_cli · test_scaffold_gha_args_forwarded_via_cli ✅
Args forwarded correctly (no unrecognized arguments error) test_scaffold_gha_args_forwarded_via_cli ✅

scaffold jenkins — Jenkins Pipeline

Command Test Result
scaffold jenkins --type build --languages python generates Jenkinsfile test_scaffold_jenkins_args_forwarded_via_cli ✅
Output contains pipeline or node block test_scaffold_jenkins_args_forwarded_via_cli ✅

scaffold gitlab — GitLab CI

Command Test Result
scaffold gitlab --type build generates .gitlab-ci.yml test_scaffold_gitlab_build · test_scaffold_gitlab_via_cli ✅
--type complete --kubernetes includes K8s deploy stage test_scaffold_gitlab_complete_with_k8s ✅
--type test --languages java generates Java test stage test_scaffold_gitlab_test_java ✅
Unified CLI output equals direct module output test_scaffold_via_cli_matches_direct_module_output ✅

scaffold argocd — ArgoCD / Flux

Command Test Result
Default generates argocd/application.yaml test_scaffold_argocd_application · test_scaffold_argocd_via_cli ✅
Generates argocd/appproject.yaml test_scaffold_argocd_appproject ✅
--allow-any-source-repo sets wildcard source in AppProject test_scaffold_argocd_appproject_allow_any_source_repo ✅
--rollouts adds Argo Rollouts fields test_scaffold_argocd_with_rollouts ✅
--method flux generates Flux CD manifests test_scaffold_argocd_flux ✅

scaffold sre — SRE Configs

Command Test Result
Generates alert rules, Grafana dashboard, SLO manifest test_scaffold_sre_all_outputs_exist ✅
Alert rules YAML has groups[].rules[] structure test_scaffold_sre_alert_rules_structure ✅
Grafana dashboard JSON contains panel definitions test_scaffold_sre_grafana_dashboard_panels ✅
SLO manifest with --slo-type latency generates latency SLO test_scaffold_sre_slo_latency ✅

scaffold devcontainer — Dev Container

Command Test Result
Default config generates devcontainer.json + devcontainer.env.json test_scaffold_devcontainer_default ✅
--languages python,go sets language env vars test_scaffold_devcontainer_languages ✅
--build-args keys written to config test_scaffold_devcontainer_build_args ✅
--kubernetes-tools k9s included in tooling section test_scaffold_devcontainer_kubernetes_tools ✅
--python-version 3.12 --go-version 1.22 propagated to env config test_scaffold_devcontainer_versions ✅
--extensions list written to VS Code extensions test_scaffold_devcontainer_extensions ✅
--forward-ports written to forwardPorts test_scaffold_devcontainer_forward_ports ✅
Reachable via unified scaffold devcontainer sub-command test_scaffold_devcontainer_via_scaffold_command ✅

scaffold cicd — Combined CI/CD (meta-scaffolder)

Command Test Result
--github --jenkins generates both GHA workflow and Jenkinsfile test_scaffold_cicd_generates_both_outputs ✅
--github only generates GHA workflow, no Jenkinsfile test_scaffold_cicd_github_only ✅
--jenkins only generates Jenkinsfile, no GHA workflow test_scaffold_cicd_jenkins_only ✅

devopsos process-first

Command Test Result
process-first --help exits 0 test_process_first_help ✅
Default (no section) prints all sections + usage footer test_process_first_all_sections · test_process_first_default_output_includes_usage_footer ✅
--section what prints principles, no footer test_process_first_section_what · test_process_first_specific_section_no_usage_footer ✅
--section mapping prints tool↔principle mapping table test_process_first_section_mapping ✅
--section tips prints learning tips test_process_first_section_tips ✅
Direct module python -m cli.process_first works test_process_first_module_direct ✅
python -m cli.process_first --section mapping works test_process_first_module_section_mapping ✅
Invalid section name gives a clean error message test_process_first_invalid_section_clean_error ✅

CLI Coverage Map

Every public CLI surface area is tested end-to-end with no real infrastructure required — all tests use temp directories and in-memory scaffolding.

CLI Surface Tests Coverage
devopsos --version / -V 3 ✅ Full
devopsos --help 1 ✅ Full
devopsos init 4 ✅ Full
devopsos scaffold (routing + help) 4 ✅ Full
devopsos scaffold gha 2 ✅ Full
devopsos scaffold jenkins 1 ✅ Full
devopsos scaffold gitlab 5 ✅ Full
devopsos scaffold argocd 5 ✅ Full
devopsos scaffold sre 4 ✅ Full
devopsos scaffold devcontainer 8 ✅ Full
devopsos scaffold cicd 3 ✅ Full
devopsos process-first 10 ✅ Full

Total: 52 tests — 0 failures — 100% of CLI commands covered

How to Run the Tests

# Install dependencies
pip install -r cli/requirements.txt pytest

# Run CLI tests
python -m pytest cli/test_cli.py -v

# Run with short failure traces
python -m pytest cli/test_cli.py -v --tb=short

See CONTRIBUTING.md for the full test philosophy and how to add new tests.