Better table parsing

[phiwi]

For now, we use various PDF-MD converters.

We will concentrate on docling, but we have to modify the scripts to

1. combine tables that span multiple pages
2. incorporate footnotes

For the current test case it's susfficient to manually re-build the table + footnotes, but future scripts should consolidate those automatically.

See:

• Part one table 22: /prj/doctoral_letters/guide/data/guidelines/docling/pdf_pages/_62/tables/table_000.json
• Part two table 22: /prj/doctoral_letters/guide/data/guidelines/docling/pdf_pages/_63/tables/table_000.json
• Footnotes: /prj/doctoral_letters/guide/data/guidelines/markdown/pages/_63.md

[emrec1]

Parsing of table section headers:

Currently the table section headers are parsed as just another table row, with class and level. Even though in some lines such as the following, the information is repeated and as such redundant:

{
      "Recommendations": "Revascularization to improve outcomes",
      "Class a": "",
      "Level b": ""
    },
    {
      "Recommendations": "In chronic coronary syndrome patients with left ventricular ejection fraction > 35%",
      "Class a": "",
      "Level b": ""
    },
    {
      "Recommendations": "In CCS patients with LVEF > 35%, myocardial revascularization is recommended, in addition to guideline-directed medical therapy, for patients with functionally significant left main stem stenosis to improve survival. 718,719,859,860",
      "Class a": "I",
      "Level b": "A"
    }

In other sections the table section header is relevant to understand the recommendation such as in:

{
      "Recommendations": "Intracoronary pressure measurement (FFR or iFR) or computation (QFR) :",
      "Class a": "",
      "Level b": ""
    },
    {
      "Recommendations": "• is recommended to guide lesion selection for intervention in patients with multivessel disease; 308,826,866,867",
      "Class a": "I",
      "Level b": "A"
    },
    {
      "Recommendations": "• should be considered at the end of the procedure to identify patients at high risk of persistent angina and subsequent clinical events; 828,830,831,868",
      "Class a": "IIa",
      "Level b": "B"
    },
    {
      "Recommendations": "• may be considered at the end of the procedure to identify lesions potentially amenable to treatment with additional PCI. 350,829,831",
      "Class a": "IIb",
      "Level b": "B"
    }

[emrec1]

Proposal:
Add the table headers as a property for each row, maybe with section header or subheader.
Example:

{
    "Section Header": "Assessment of procedural risks and post-procedural outcomes",
    "Sub Header": "Intracoronary pressure measurement (FFR or iFR) or computation (QFR) :",
    "Recommendations": "• is recommended to guide lesion selection for intervention in patients with multivessel disease; 308,826,866,867",
    "Class a": "I",
    "Level b": "A"
},
{
    "Section Header": "Assessment of procedural risks and post-procedural outcomes",
    "Sub Header": "Intracoronary pressure measurement (FFR or iFR) or computation (QFR) :",
    "Recommendations": "• should be considered at the end of the procedure to identify patients at high risk of persistent angina and subsequent clinical events; 828,830,831,868",
    "Class a": "IIa",
    "Level b": "B"
},
{
    "Section Header": "Assessment of procedural risks and post-procedural outcomes",
    "Sub Header": "Intracoronary pressure measurement (FFR or iFR) or computation (QFR) :",
    "Recommendations": "• may be considered at the end of the procedure to identify lesions potentially amenable to treatment with additional PCI. 350,829,831",
    "Class a": "IIb",
    "Level b": "B"
}

[emrec1]

Parsing error, Properties Class a, and Level b are missing.
Section header is used as a Property Label: "Choice of revascularization modality"

Part two table 22: /prj/doctoral_letters/guide/data/guidelines/docling/pdf_pages/_63/tables/table_000.json

{
  "table_id": 0,
  "source_file": "/home/pwiesenbach/CardioGuidelinesGraph/scripts_emre/data/guidelines/pdf_pages/_63.pdf",
  "caption": [],
  "data": [
    {
      "Choice of revascularization modality": "It is recommended that physicians select the most appropriate revascularization modality based on patient profile, c coronary anatomy, d procedural factors, e LVEF, preferences, and outcome expectations. 719,725,728,792-795,801,816,820,822,859,869",
      "2024": "C © ESC"
    }
  ],
  "num_rows": 1,
  "num_cols": 2
}

[phiwi]

It's yours now :-)

[emrec1]

Will switch to pdfplumber and md parsing.

Find manually cleaned table 22 at /prj/doctoral_letters/guide/data/evaluation/table_22_manual_clean_headers.json

Added the Table Header and Section Header for each Row, optionally added a Sub Header if it existed.

Example from json:

{
{
    "Table Header": "Recommendations for revascularization in patients with chronic coronary syndrome",
    "Section Header": "Assessment of procedural risks and post-procedural outcomes",
    "Recommendations": "Intracoronary imaging guidance by IVUS or OCTis recommended when performing PCI on anatomically complex lesions, in particular left main stem, true bifurcations, and long lesions. 866,337,810,840,841",
    "Class a": "I",
    "Level b": "A"
},
{
    "Table Header": "Recommendations for revascularization in patients with chronic coronary syndrome",
    "Section Header": "Assessment of procedural risks and post-procedural outcomes",
    "Sub Header": "Intracoronary pressure measurement (FFR or iFR) or computation (QFR) :",
    "Recommendations": "• is recommended to guide lesion selection for intervention in patients with multivessel disease; 308,826,866,867",
    "Class a": "I",
    "Level b": "A"
}
}

[emrec1]

Switched to camelot, did not work, in the end switched to pymupdf block parsing, here we have good results, Recommendation Tables are parsed successfully, currently working on edge cases like adding footnotes and tables that span multiple pages.

[emrec1]

all recommendation tables parsed completely, and json tables manually checked for mistakes.
Path to Folder: /prj/doctoral_letters/guide/data/parsing_tables/checked_rec_tables

List of manual edits:

• 2 | odd subheader
• 10 | added continued
• 11 | removed footnotes
• 12 | odd subheaders
• 16 | added continued
• 20 | section header set to  null
• 23 | odd subheader
• 24 | added two missing continued
• 25 | added 3 missing continued
• 26 | added missing row
• 29 | section header incomplete

other tables and footnotes not included