Requirement annotation search needs evidence-only mode (exclude docs, fenced examples, top-level declarations)

[buger]

Problem

probe search --allow-tests --exact --no-merge -o json <REQ-ID> <root> is close to what ReqProof needs for live MC/DC witness discovery, but after upgrading to @probelabs/probe@0.6.0-rc314 it still does not provide a clean "real witness/test blocks only" mode.

rc314 added is_test to JSON output, which helps, but the search still returns mixed results that force downstream tools to maintain a large amount of filtering logic.

What we need

For requirement-annotation search, probe should support a mode that returns only source blocks that are valid requirement evidence candidates, especially test/witness blocks.

In practice we want all of the following:

1. Include real test / witness code blocks.
2. Exclude docs and help files.
3. Exclude fenced code examples from docs.
4. Exclude top-level implementation declarations that merely carry file-level requirement comments.
5. Prefer the owning symbol/block over generic surrounding declaration noise.

This should work across languages. The requirement comments are not Go-specific.

Current command

probe search --allow-tests --exact --no-merge -o json SYS-REQ-985 .

Environment used:

• package: @probelabs/probe@0.6.0-rc314
• CLI: probe-code 0.6.0

Actual output problem

The results still include all of these categories together:

1. Good results we want

Actual test functions:

• pkg/workflow/workflow_code_mcdc_test.go
  • TestFormatNestedCodeMCDCProgressSummarizesSharedTestCommandWhenNotVerbose
  • TestSharedCodeMCDCTestRunIsReusedByCoverageCheck

These now include is_test: true, which is useful.

2. Docs/help blocks we do not want

Examples from:

• docs/help/checks/code_mcdc_coverage.md
• docs/help/checks/slow_tests_clean.md
• docs/help/commands/mcdc.md

This also includes fenced examples, for example a Markdown code fence containing:

// Verifies: SYS-REQ-985, SYS-REQ-986
// MCDC SYS-REQ-985: code_mcdc_audit_enabled=F, shared_test_evidence_run=F => TRUE
func TestSharedWorkflowBehavior(t *testing.T) {
    t.Parallel()
}

That block is returned with:

• node_type: "fenced_code_block"
• is_test: true

So is_test alone is not enough for consumers.

3. Top-level implementation declarations we do not want

These are still returned as requirement hits even though they are not test/witness blocks:

• pkg/workflow/workflow.go
• pkg/workflow/workflow_code_mcdc.go

Those hits come from file-level requirement comments and top-level declarations such as type ... struct{}.

Why this matters

ReqProof is using probe as the backend for live requirement witness discovery. We want source-derived evidence to be evaluated in real time instead of relying on stale regenerated caches.

Right now we still need downstream filtering logic roughly like this:

• skip docs/, specs/, .proof/, vendor/, node_modules/
• skip fenced_code_block
• require either:
  • explicit witness comments, or
  • test-like file / symbol heuristics
• ignore top-level declarations that only carry broad requirement comments

That works, but it means probe is still not giving us a first-class "requirement evidence blocks" query mode.

Desired solution

Any of the following would solve the problem well:

Option A: add a focused search flag

Something like one of these:

• --evidence-only
• --test-evidence-only
• --requirement-blocks-only

Behavior:

• include real source blocks that are plausible requirement evidence carriers
• exclude markdown/docs/fenced code/help examples
• exclude top-level/module-level declaration noise
• keep multi-language support

Option B: add stronger structural metadata to JSON

If probe does not want a new filter mode yet, then JSON output should expose enough to let downstream tools do this reliably without brittle heuristics. For example:

• scope: file | module | declaration | function | test | example
• is_doc: bool
• is_example: bool
• is_fenced_example: bool
• has_requirement_annotation: bool
• stable owner/parent identity for the logical block

The new is_test field in rc314 is a good start, but it is not sufficient because fenced Markdown examples can also be marked is_test: true.

Option C: make test/evidence ownership explicit

When a requirement annotation is attached to a comment above a block, return the owning block in a way that lets consumers request only that owning block and not unrelated broader declaration/file context.

Concrete expected behavior for this repro

For:

probe search --allow-tests --exact --no-merge -o json SYS-REQ-985 .

an evidence-focused mode should ideally return only these two blocks from this repo:

• pkg/workflow/workflow_code_mcdc_test.go:178
• pkg/workflow/workflow_code_mcdc_test.go:200

and not return:

• docs/help markdown
• fenced code examples
• pkg/workflow/workflow.go
• pkg/workflow/workflow_code_mcdc.go

Relationship to issue #553

Issue #553 helped clarify duplicate/block-shape behavior. This issue is the next step: even with rc314 and is_test, we still need a first-class way to ask probe for the real requirement evidence blocks only.

[buger]

One clarification to make this broadly useful beyond ReqProof: if probe exposes structural metadata instead of or before adding a dedicated filter flag, the key fields for downstream tools would be stable owner/symbol identity and source location. Concretely, fields like owner symbol/function/class name, canonical path, line range, stable block ID, and an explicit scope/kind would let many tools consume requirement-annotation results without custom heuristics. This would help traceability tools, code intelligence tools, test-mapping tools, and documentation/linking consumers, not just MC/DC witness discovery.

[probelabs]

Issue Analysis: Evidence-Only Mode for Requirement Annotation Search

Thank you for this detailed and well-structured issue report. I understand the core problem: ReqProof needs a clean "requirement evidence blocks only" mode that returns actual test/witness code while excluding documentation, fenced examples, and top-level declarations that merely carry requirement comments.

What I Understand

You're using probe search --allow-tests --exact --no-merge -o json <REQ-ID> <root> for live MC/DC witness discovery. While rc314 added the is_test field to JSON output, the search still returns mixed results:

✅ What you want:

• Real test functions (e.g., TestFormatNestedCodeMCDCProgressSummarizesSharedTestCommandWhenNotVerbose)
• Actual witness code blocks

❌ What you don't want:

• Documentation files (docs/help/checks/*.md)
• Fenced code examples in Markdown (marked as is_test: true but are just examples)
• Top-level declarations that only carry file-level requirement comments (e.g., type ... struct{})

Current Implementation Analysis

Based on my analysis of the codebase, here's what's happening:

1. is_test Field Limitations

The is_test field in JSON output is computed from two sources (search_output.rs:620-750):

let file_is_test = is_test_file(Path::new(&r.file));
let code_is_test = !file_is_test && is_test_code_block(&r.code, &r.node_type);

The problem: Markdown fenced code blocks can also be marked is_test: true (markdown.rs:136-176):

fn is_test_node(&self, node: &Node, source: &[u8]) -> bool {
    // Check for code blocks with test-related languages
    if node.kind() == "fenced_code_block" {
        if let Some(lang) = self.get_code_block_language(node, source) {
            let is_test_lang = lower_lang.contains("test")
                || lower_lang.contains("spec")
                || lower_lang == "jest"
                || lower_lang == "mocha"
                || lower_lang == "pytest";
            return is_test_lang;
        }
    }
}

This means a Markdown code fence with go test will be marked is_test: true, even though it's just documentation.

2. No Evidence-Only Filter Exists

Currently, probe has these filters (search_options.rs):

• allow_tests: Include/exclude test files
• exclude_filenames: Skip filename matching
• Custom ignore patterns

But there's no mechanism to distinguish:

• Real source files vs. documentation
• Fenced examples vs. actual test code
• Top-level declarations vs. test functions

3. File Path Filtering is Manual

Probe does have file filtering capabilities (file_paths.rs:1112-1130), but it requires manual ignore patterns. There's no built-in "exclude docs/" logic.

Proposed Solutions

Option A: Add --evidence-only Flag (Recommended)

This would be the cleanest solution for ReqProof's use case. Implementation would involve:

1. Add new search option in cli.rs and search_options.rs
2. Filter logic in file_processing.rs to exclude:
   • Files matching documentation patterns (docs/, *.md, specs/, .proof/)
   • fenced_code_block node types
   • Top-level declarations without test-like content
3. JSON metadata to indicate evidence-only mode was used

Pros:

• First-class support for your use case
• No downstream filtering needed
• Clear, explicit intent

Cons:

• Requires new feature development
• May need tuning for different project structures

Option B: Enhanced JSON Metadata

Add structural metadata to JSON output (search_output.rs:620-750):

{
  "file": "pkg/workflow/workflow_code_mcdc_test.go",
  "lines": [178, 200],
  "node_type": "function_declaration",
  "is_test": true,
  "scope": "function",        // NEW: file | module | function | test | example
  "is_doc": false,              // NEW
  "is_example": false,          // NEW
  "is_fenced_example": false,   // NEW
  "has_requirement_annotation": true,  // NEW
  "owning_symbol": "TestFormatNestedCodeMCDC..."  // NEW
}

Pros:

• Flexible for different use cases
• No new CLI flags needed
• Downstream tools can implement custom logic

Cons:

• Requires downstream filtering logic (what you're trying to avoid)
• More complex JSON schema

Option C: Hybrid Approach

Implement both:

1. Add --evidence-only flag for simple cases
2. Add enhanced metadata for advanced filtering

This gives you a simple solution now while enabling future flexibility.

Clarifying Questions

To help prioritize and implement the right solution:

1. File patterns: Are the documentation patterns consistent across your projects? (e.g., always docs/, specs/, .proof/, vendor/, node_modules/)

2. Top-level declarations: Should --evidence-only exclude ALL top-level type/struct declarations, or only those without test functions?

3. Language support: Do you need this to work across all languages probe supports, or primarily Go?

4. Fenced examples: Should we exclude ALL fenced code blocks in Markdown, or only those with test-related language tags?

5. Priority: Would you prefer a quick fix (Option B metadata) or a complete solution (Option A flag)?

Recommended Next Steps

1. Start with Option B (enhanced metadata) as it's less invasive and can be shipped faster
2. Add Option A (--evidence-only flag) as a follow-up enhancement
3. Document the recommended ignore patterns for ReqProof users in the meantime

Workaround for Now

Until this is implemented, you can use probe's existing ignore patterns:

probe search --allow-tests --exact --no-merge -o json \
  --ignore 'docs/*' --ignore 'specs/*' --ignore '.proof/*' \
  --ignore 'vendor/*' --ignore 'node_modules/*' \
  SYS-REQ-985 .

Then filter results downstream to exclude node_type: "fenced_code_block".

---

Issue Classification: enhancement
Priority: medium (clear use case, but workarounds exist)
Complexity: moderate (requires new filtering logic and metadata)
Recommended Labels: enhancement, feature, documentation

References

• probe/src/search/search_output.rs:620-750 – JSON output formatting with is_test field
• probe/src/language/markdown.rs:136-176 – Markdown test node detection (fenced code blocks)
• probe/src/search/search_options.rs – Search options structure
• probe/src/cli.rs:189-197 – CLI argument definitions
• probe/src/search/file_processing.rs:62-80 – File processing parameters
• probe/src/extract/file_paths.rs:1112-1130 – File path filtering logic

---

Powered by Visor from Probelabs

💡 TIP: You can chat with Visor using /visor ask <your question>