Analyze procedure mis-mapping report: align Ground Truth and model labeling

[phiwi]

Hi team,

Could we schedule a focused review of the current mis-mapping findings in:
https://github.com/dieterich-lab/CardioGuidelinesGraph/blob/master/docs/trackers/grounding/procedure_mis_mapping_report.md

The goal is to understand why Ground Truth and model labeling are still not fully synchronized and define concrete fixes.

Proposed To-Dos

1. Review disagreement cases and classify each mismatch into one of these buckets:

• Ground Truth likely incorrect or outdated.
• Model labeling likely incorrect.
• Task / label semantics ambiguous (needs rule clarification).

2. For each bucket, capture:

• Representative examples.
• Root-cause hypothesis.
• Required action (data correction, prompt/rule update, ontology change, or evaluation-policy change).

3. Define a short remediation plan with owners and expected impact:

• Quick wins (can be fixed immediately).
• Medium-term fixes (pipeline or labeling policy changes).
• Validation plan for confirming reduced mismatch rate.

4. Re-check SNOMED subset dependency (important reminder):

• Verify whether we are still using the previously created cardiology-oriented SNOMED subset.
• If this subset step/file was dropped or bypassed, re-create and reinstate the cardiology-focused SNOMED subset as a priority.
• Document where this subset is generated, stored, and consumed in the current pipeline.

Desired Outcome

• A clear decision per mismatch type (GT fix vs model fix vs ambiguity handling).
• A tracked action list with owners.
• Confirmed status of cardiology SNOMED subset usage, with re-creation task initiated if needed.

Thanks everyone. This should help us stabilize labeling quality and speed up SNOMED grounding.

[phiwi]

Quick forensic summary + recovery proposal:

Findings

• Current SNOMED grounding/search in src/cardio_graph_core/snomedct/snomed_query.py is not constrained to a cardiology subset.
• Current vector ingest in src/cardio_graph_core/neo4j/snomed_vector_ingest.py embeds from the global description table (active terms), i.e. whole-DB scope.
• The old cardiology subset tooling existed in:
  • src/cardio_graph/snomedct_utils/ontology_config.yaml
  • src/cardio_graph/snomedct_utils/generate_cardio_ontology.py
• Timeline:
  • subset YAML removed in commit 13d69d3
  • legacy SNOMED tooling moved to archive in commit e97afd5
  • archive later removed (commit 4a55dd6)

Recovery proposal (implementing now)

1. Restore the historical ontology YAML (as active config under cardio_graph_core).
2. Restore subset-creation script (modernized imports to cardio_graph_core).
3. Generate a fresh cardiology subset artifact (concept-id allowlist + ontology output).
4. Add subset filter support to SNOMED vector ingest CLI.
5. Wipe neo4j-dev3 and re-ingest vectors using only the subset concept IDs.

Expected outcome

• Smaller, cardiology-focused vector space
• Faster indexing/ingest
• Better grounding precision via reduced semantic noise

[phiwi]

Status update: ontology construction is underway (Slurm job in progress) using the recovered cardiology subset pipeline.

Essential commits on master:

• 06febb8
• 23c0a0c

I will post follow-up metrics and schema validation once generation completes.

[phiwi]

Status snapshot (research + implementation)

1. Recovery and implementation: completed

• Recovered and reintroduced the cardiology subset generation pipeline in active core paths.
• Added subset-aware vector ingest filtering.
• Split and pushed implementation in focused commits.

2. Evaluation and tuning: in progress

• First full GT-seeded run completed successfully with 100% GT seed coverage.
• Purity audit showed strong recall but too much off-domain drift, mainly from parent expansion.
• Applied first targeted tuning change:
  • parent depth default set to 0
  • child depth kept at 1

3. Current experiment

• New subset-generation run is currently running with:
  • seed_mode=both
  • parent_depth=0
  • child_depth=1
• Objective: measure whether drift drops while GT coverage remains intact.

Next step after completion

• Re-run the same sensitivity/specificity proxy audit on the new outputs.
• Compare before vs after and decide whether to tighten seed terms and/or expansion strategy further.

[phiwi]

Quick update: pushed additional subset tuning to reduce lexical noise in seed terms (commit: ecab2c6).

A new tuned generation run is currently in progress (job 629543); I’ll post the before/after specificity/sensitivity comparison once it completes.

[phiwi]

Update after tuned run 629543 (parent_depth=0, child_depth=1 + lexical seed pruning) completed.

Health / quality summary

• Job: COMPLETED (exit 0)
• Final subset size: 19,018 concepts
• Seed term IDs: 10,395
• GT seed coverage: 69/69 (100%)

Before/after quality trend (same audit method, n=200 sample)

1. Older broad run (parent=2, child=1)

• subset size: 31,655
• likely-cardio: 38.0%
• off-domain: 54.0%
• likely-cardio+maybe: 46.0%

2. First tuned run (parent=0, child=1)

• subset size: 23,238
• likely-cardio: 54.0%
• off-domain: 40.0%
• likely-cardio+maybe: 60.0%

3. Current tuned run (parent=0, child=1 + seed pruning)

• subset size: 19,018
• likely-cardio: 67.5%
• off-domain: 22.0%
• likely-cardio+maybe: 78.0%
• likely-cardio 95% CI: [60.7%, 73.6%]

Takeaway

• Sensitivity proxy (GT coverage) stayed intact at 100%.
• Specificity proxy improved substantially across both tuning steps.
• Parent expansion removal + targeted lexical cleanup appears to be the right direction.

[phiwi]

Korrektur/Präzisierung zum vorherigen Kommentar (Formatting-Artefakt):

Kurzes Update nur zum neuen Run (ohne Wiederholung der bisherigen Schritte):

• Neuer Tuning-Run wurde gestartet: Job 629771 (cardio-subset-gen)
• Aktueller Status: RUNNING
• Dieser Run testet gezielte Recall-Erweiterungen über neue Seeds für bisherige Lücken, insbesondere:
  • pericarditis
  • myocarditis
  • aortic stenosis
  • mitral regurgitation
  • papillary muscle
  • supravalvar
• Guardrails im Run bleiben bewusst stabil: seed_mode=both, parent_depth=0, child_depth=1

Sobald der Job fertig ist, poste ich nur die entscheidenden Ergebnispunkte (Subset-Größe, GT-Coverage, Sensitivity/Specificity-Delta und Keep/Revert-Empfehlung).

[phiwi]

Quick update: we have started a fresh automated vector-grounding evaluation run against the new cardio-focused SNOMED subset index (instead of the full 1.5M+ search space). We are now on the way to generating updated automated SNOMED mappings; results should be ready for analysis soon once the run finishes and artifacts are written.

[phiwi]

Update: proposed operating design to stabilize evaluation and collaboration

We now run a strict dual-track framework:

1. Scientific track (benchmark/generalization):

• locked_test split
• rescue disabled
• primary number for scientific comparison

2. Productive track (deployment quality):

• locked_test split
• deterministic rescue map derived only from non-holdout (train/dev) evidence
• one-shot holdout evaluation after map freeze

Current canonical mappings document for colleague review:
https://github.com/dieterich-lab/CardioGuidelinesGraph/blob/master/docs/reference/grounding/current_snomed_mappings_scientific_track.md

@emrec1 please compare your selected ground truth entries against the current system mappings in that document and classify each disagreement as:

• GT correction needed
• model mapping correction needed
• semantic/policy ambiguity (needs rule clarification)

Please add representative examples per bucket so we can turn them into tracked remediation tasks.