Add support for pointing indexes in ocaml-index by Tim-ats-d · Pull Request #2051 · ocaml/merlin

Tim-ats-d · 2026-03-24T13:19:50Z

This PR adds support for pointing to an index file that already exists on disk.

This mechanism results in significantly smaller index files and fewer write operations.

art-w

@voodoos raised another issue: Since the indexes are not self-contained anymore, it's possible that a pointing index targets another index file that has been removed or updated. If we don't check for that and follow pointers to arbitrary locations, some very bad things could happen!

Tim-ats-d · 2026-03-31T09:41:31Z

@voodoos raised another issue: Since the indexes are not self-contained anymore, it's possible that a pointing index targets another index file that has been removed or updated. If we don't check for that and follow pointers to arbitrary locations, some very bad things could happen!

I think I could make some kind of defensive programming and simply fail if the pointed-to index file no longer exists. What do you think?

… expired data.

art-w

It's looking good! I only have some minor requests to improve the id checking logic :)

Tim-ats-d · 2026-04-03T12:52:18Z

Here are some benchmark results from Hyperfine I got running ocaml-index on a bunch of ocaml-index files generated from Merlin repo.

Without pointing index:

Time (mean ± σ):      2.425 s ±  0.087 s    [User: 1.968 s, System: 0.395 s]
Range (min … max):    2.313 s …  2.553 s    10 runs

With pointing-index enabled:

Time (mean ± σ):     896.2 ms ±  42.5 ms    [User: 640.6 ms, System: 245.2 ms]
Range (min … max):   840.6 ms … 972.3 ms    10 runs

Complete command I used:

ocaml-index aggregate dot_merlin_reader.ocaml-index merlin_analysis.ocaml-index merlin_commands.ocaml-index merlin_config.ocaml-index merlin_dot_protocol.ocaml-index merlin_extend.ocaml-index merlin_index_format.ocaml-index merlin_kernel.ocaml-index merlin_specific.ocaml-index merlin_utils.ocaml-index ocaml-index.ocaml-index ocaml_compression.ocaml-index ocaml_parsing.ocaml-index ocaml_preprocess.ocaml-index ocaml_typing.ocaml-index ocaml_utils.ocaml-index ocamlmerlin_server.ocaml-index os_ipc.ocaml-index query_commands.ocaml-index query_protocol.ocaml-index  --root . --rewrite-root

Tim-ats-d · 2026-04-03T13:39:05Z

Concerning reading performance, here are some other metrics. I ran ocaml-index dump on a project ocaml-index file aggregating the same files used on the previous benchmark.

With pointing-index enabled:

Time (mean ± σ):      3.990 s ±  0.076 s    [User: 3.372 s, System: 0.581 s]
  Range (min … max):    3.909 s …  4.129 s    10 runs

Without:

Time (mean ± σ):      3.803 s ±  0.141 s    [User: 3.417 s, System: 0.356 s]
  Range (min … max):    3.628 s …  4.092 s    10 runs

So reading performances are mostly similar.

…opened.

Tim-ats-d · 2026-04-08T12:20:49Z

My last commit should address your two last review comments @art-w

art-w

Thanks! I left some additional comments because I'm still not convinced that the store id validation is correct... In fact it might be easier to add the id field in the store type directly (instead of On_disk), since it's as relevant as the filename to identify the expected index file.
I also did not see the code to report a message to users that their build is out of date? (when we run into an invalid store file)

…store`.

… file.

…error message.

art-w

Thanks a lot, this looks great! I have some minor remarks but nothing blocking, in case @voodoos has time for another look :)

art-w · 2026-04-10T11:25:17Z

  | Serialized of { loc : int }
  | Serialized_reused of { loc : int }
  | On_disk of { store : store; loc : int; schema : 'a schema }
+  | On_disk_ptr of { filename : string; loc : int; id : int }


(cc @voodoos) Do we know if the filenames are absolute or relative? (I wonder if there's a risk that ocaml-index could be called from different folders, such that the pointed filenames are later not found because of the relative paths interpretation)

There are relative. We can maybe use Unix.realpath on such filename.

The filenames in the locations stored in the indexes are relative unless --root is passed to the indexer.
In dune they should always be relative to the root of the build context (like _build/default).

voodoos · 2026-04-10T14:14:10Z

  | Serialized of { loc : int }
  | Serialized_reused of { loc : int }
  | On_disk of { store : store; loc : int; schema : 'a schema }
+  | On_disk_ptr of { filename : string; loc : int; id : int }


The filenames in the locations stored in the indexes are relative unless --root is passed to the indexer.
In dune they should always be relative to the root of the build context (like _build/default).

voodoos

Thanks, looks fine to me. Just one worry about the new exception handling when reading the indexes for occurrences. Especially since it's a point in time were indexes are much more likely to be out of sync than during aggregation at build time.

Also, it's all or nothing, but for user queries it would be great to still be able to provide partial results if some of the files are out-of-sync. Would that be hard to do ? If so let's just add a comment about it around the place where we read indexes for occurrences.

voodoos · 2026-04-21T10:30:35Z

And this definitely deserves a changelog entry.

This reverts commit 853a38b.

…lated features.

voodoos · 2026-04-21T13:55:10Z

Thanks, can you rebase ? there are some merge conflicts.

Tim-ats-d added 2 commits March 23, 2026 17:00

Progress.

acc2ec6

Clean up.

c4a34e4

art-w suggested changes Mar 24, 2026

View reviewed changes

Comment thread src/index-format/granular_marshal.ml Outdated

Use shared cache sync with filesystem for ocaml-index file reading.

7567b8f

voodoos reviewed Mar 30, 2026

View reviewed changes

Comment thread src/index-format/granular_marshal.ml Outdated

voodoos reviewed Mar 30, 2026

View reviewed changes

Comment thread src/index-format/granular_marshal.ml Outdated

Use shared cache in Granular_marshal.read.

75cf152

Tim-ats-d force-pushed the pointing-indexes branch 2 times, most recently from 0908c3a to 8032c54 Compare April 1, 2026 13:21

Attach a random number to each index file in order to prevent reading…

bc62cca

… expired data.

Tim-ats-d force-pushed the pointing-indexes branch from 8032c54 to bc62cca Compare April 1, 2026 13:26

More precise error message.

a8ee6a6

art-w suggested changes Apr 3, 2026

View reviewed changes

Tim-ats-d added 3 commits April 3, 2026 15:07

Use 8 bytes to store index file random ID.

8411fc7

Make is_on_disk behavior correct.

d8c00da

Use a random state instead of initialize a global one.

38b82cd

Delay pointing index check and check if a file descriptor is already …

c9b9bf3

…opened.

art-w suggested changes Apr 9, 2026

View reviewed changes

Tim-ats-d added 3 commits April 9, 2026 14:03

Store ID in type store and delay the index ID check in `force_open_…

01271f8

…store`.

Proper exception raising when attempting to consult an outdated index…

e56a469

… file.

Catch Outdated_store exception in ocaml-index binary and provides an …

dbde744

…error message.

art-w approved these changes Apr 10, 2026

View reviewed changes

voodoos reviewed Apr 10, 2026

View reviewed changes

Tim-ats-d added 2 commits April 13, 2026 13:22

Last small fixes.

1c6b067

Check store filename too in force_open_store.

aba0c96

art-w reviewed Apr 20, 2026

View reviewed changes

Comment thread src/ocaml-index/bin/ocaml_index.ml

Add some tests to ensure pointing index behavior in case of errors.

9a78a18

voodoos approved these changes Apr 21, 2026

View reviewed changes

Comment thread src/index-format/granular_marshal.mli Outdated

Comment thread src/ocaml-index/bin/ocaml_index.ml

Comment thread tests/test-dirs/occurrences/project-wide/union.t Outdated

Tim-ats-d added 6 commits April 21, 2026 14:37

Fix typo in doc.

c30495d

Add a best effort mode to allow reading outdated index files.

853a38b

Add change entry.

855a040

Revert "Add a best effort mode to allow reading outdated index files."

0435c36

This reverts commit 853a38b.

Labelled "id" parameter.

ab3743d

Guard index reading against Outdated_store exception in occurences re…

c3a3bee

…lated features.

Merge branch 'main' into pointing-indexes

9fe30a3

Conversation

Tim-ats-d commented Mar 24, 2026

Uh oh!

art-w left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Tim-ats-d commented Mar 31, 2026

Uh oh!

art-w left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Tim-ats-d commented Apr 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Tim-ats-d commented Apr 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Tim-ats-d commented Apr 8, 2026

Uh oh!

art-w left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

art-w left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

art-w Apr 10, 2026

Choose a reason for hiding this comment

Uh oh!

Tim-ats-d Apr 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

voodoos Apr 10, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

voodoos Apr 10, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

voodoos left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

voodoos commented Apr 21, 2026

Uh oh!

voodoos commented Apr 21, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Tim-ats-d commented Apr 3, 2026 •

edited

Loading

Tim-ats-d commented Apr 3, 2026 •

edited

Loading

Tim-ats-d Apr 10, 2026 •

edited

Loading

voodoos left a comment •

edited

Loading