Skip to content

Commit 9f3b664

Browse files
committed
fix(search): column bulk operations search not returning results at scale
When searching by column name pattern (e.g., "MAT") in column bulk operations, the composite aggregation returned ALL column names from matching documents, then post-filtered in Java. With 20000+ columns, the first composite page of 25 names rarely contained matches, so users saw 0 results. Switch to terms aggregation with `include` regex when a search pattern is set. This filters at the ES/OS aggregation level — only matching column names produce buckets. Two-phase approach: (1) lightweight names query to get all matching names + accurate total, (2) targeted data query with top_hits for the current page only.
1 parent af7ffe7 commit 9f3b664

File tree

4 files changed

+686
-197
lines changed

4 files changed

+686
-197
lines changed

openmetadata-service/src/main/java/org/openmetadata/service/search/ColumnAggregator.java

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -21,6 +21,28 @@ public interface ColumnAggregator {
2121

2222
ColumnGridResponse aggregateColumns(ColumnAggregationRequest request) throws IOException;
2323

24+
/**
25+
* Convert a plain text pattern to a case-insensitive regex for ES/OS terms include. Lucene regex
26+
* does not support (?i), so each letter is expanded to a character class: "MAT" → [mM][aA][tT].
27+
*/
28+
static String toCaseInsensitiveRegex(String pattern) {
29+
StringBuilder sb = new StringBuilder(".*");
30+
for (char c : pattern.toCharArray()) {
31+
if (Character.isLetter(c)) {
32+
sb.append('[')
33+
.append(Character.toLowerCase(c))
34+
.append(Character.toUpperCase(c))
35+
.append(']');
36+
} else if (".+*?|[](){}^$\\~@&#<>\"".indexOf(c) >= 0) {
37+
sb.append('\\').append(c);
38+
} else {
39+
sb.append(c);
40+
}
41+
}
42+
sb.append(".*");
43+
return sb.toString();
44+
}
45+
2446
class ColumnAggregationRequest {
2547
private int size = 1000;
2648
private String cursor;

0 commit comments

Comments
 (0)