Commit a773a85
fix(search): column bulk operations search not returning results at scale
When searching by column name pattern (e.g., "MAT") in column bulk
operations, the composite aggregation returned ALL column names from
matching documents, then post-filtered in Java. With 20000+ columns,
the first composite page of 25 names rarely contained matches, so
users saw 0 results.
Switch to terms aggregation with `include` regex when a search pattern
is set. This filters at the ES/OS aggregation level — only matching
column names produce buckets. Two-phase approach: (1) lightweight
names query to get all matching names + accurate total, (2) targeted
data query with top_hits for the current page only.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>1 parent af7ffe7 commit a773a85
File tree
4 files changed
+686
-197
lines changed- openmetadata-service/src
- main/java/org/openmetadata/service/search
- elasticsearch
- opensearch
- test/java/org/openmetadata/service/search
4 files changed
+686
-197
lines changedLines changed: 22 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
21 | 21 | | |
22 | 22 | | |
23 | 23 | | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
24 | 46 | | |
25 | 47 | | |
26 | 48 | | |
| |||
0 commit comments