[ISSUE #9254] Add CombineConsumeQueue to support CQ migration by qianye1001 · Pull Request #9256 · apache/rocketmq

qianye1001 · 2025-03-17T12:56:08Z

Which Issue(s) This PR Fixes

Brief Description

How Did You Test This Change?

codecov-commenter · 2025-03-18T02:06:29Z

Codecov Report

Attention: Patch coverage is 51.68869% with 329 lines in your changes missing coverage. Please review.

Project coverage is 48.05%. Comparing base (c9ea749) to head (5620d38).
Report is 31 commits behind head on develop.

Files with missing lines	Patch %	Lines
...rocketmq/store/queue/CombineConsumeQueueStore.java	41.15%	140 Missing and 13 partials ⚠️
...apache/rocketmq/store/queue/ConsumeQueueStore.java	50.00%	77 Missing and 6 partials ⚠️
...rocketmq/store/queue/RocksDBConsumeQueueStore.java	47.05%	26 Missing and 1 partial ⚠️
...etmq/store/metrics/RocksDBStoreMetricsManager.java	0.00%	16 Missing ⚠️
...pache/rocketmq/store/dledger/DLedgerCommitLog.java	41.17%	6 Missing and 4 partials ⚠️
...main/java/org/apache/rocketmq/store/CommitLog.java	59.09%	5 Missing and 4 partials ⚠️
...ache/rocketmq/store/queue/RocksDBConsumeQueue.java	74.19%	8 Missing ⚠️
...ocketmq/broker/processor/AdminBrokerProcessor.java	0.00%	6 Missing ⚠️
...org/apache/rocketmq/store/DefaultMessageStore.java	89.58%	1 Missing and 4 partials ⚠️
...ache/rocketmq/store/config/MessageStoreConfig.java	75.00%	5 Missing ⚠️
... and 4 more

Additional details and impacted files

@@              Coverage Diff              @@
##             develop    #9256      +/-   ##
=============================================
+ Coverage      47.96%   48.05%   +0.08%     
- Complexity     11908    11982      +74     
=============================================
  Files           1307     1308       +1     
  Lines          92021    92166     +145     
  Branches       11775    11789      +14     
=============================================
+ Hits           44139    44288     +149     
+ Misses         42401    42375      -26     
- Partials        5481     5503      +22

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

lollipopjin · 2025-04-22T09:10:02Z

        }
    }
+
+    public void initializeWithOffset(long offset) {


This is used for mount --> main in container mode? If so, add some java docs here.

lollipopjin · 2025-04-22T09:13:12Z

+
+            // update the max and min offset
+            if (offset > 0) {
+                this.consumeQueueStore.updateCqOffset(topic, queueId, 0L, offset - 1, true);


How about use two methods for updating CQ offset for min and max offsets.

lizhimins · 2025-05-29T06:33:20Z

+        final String storePath,
+        final int mappedFileSize,
+        final MessageStore messageStore,
+        final ConsumeQueueStore consumeQueueStore) {


This format looks a bit strange

lizhimins · 2025-05-29T07:22:41Z

+
+        currentReadStore = getInnerStoreByString(messageStoreConfig.getCombineCQPreferCQType());
+        if (currentReadStore == null) {
+            log.error("CombineConsumeQueue choosePreferCQ fail, prefer={}", messageStoreConfig.getCombineCQPreferCQType());


日志的风格，前后不太一致

lollipopjin

LGTM now.

qianye1001 marked this pull request as draft March 17, 2025 12:56

qianye1001 changed the title ~~[ISSUE #9254] Refactor DefaultMessageStorage type/enhancement~~ [ISSUE #9254] Refactor DefaultMessageStorage Mar 18, 2025

qianye1001 marked this pull request as ready for review March 18, 2025 03:31

qianye1001 changed the title ~~[ISSUE #9254] Refactor DefaultMessageStorage~~ [ISSUE #9254] Refactor CQ-related thread in DefaultMessageStorage Mar 18, 2025

qianye1001 changed the title ~~[ISSUE #9254] Refactor CQ-related thread in DefaultMessageStorage~~ [ISSUE #9254] Refactor CQ-related in DefaultMessageStorage Mar 19, 2025

qianye1001 force-pushed the Refactor-DefaultMessageStore branch 2 times, most recently from ad190f2 to 61ee2f4 Compare March 25, 2025 07:03

qianye1001 force-pushed the Refactor-DefaultMessageStore branch 2 times, most recently from dc014b5 to a8996fa Compare April 10, 2025 07:03

lizhimins reviewed Apr 16, 2025

View reviewed changes

Comment thread store/src/main/java/org/apache/rocketmq/store/CommitLog.java Outdated

qianye1001 force-pushed the Refactor-DefaultMessageStore branch from da78405 to 59ee17a Compare April 18, 2025 03:33

lizhimins reviewed Apr 18, 2025

View reviewed changes

Comment thread store/src/main/java/org/apache/rocketmq/store/queue/CombineConsumeQueueStore.java Outdated

lizhimins reviewed Apr 18, 2025

View reviewed changes

Comment thread store/src/main/java/org/apache/rocketmq/store/queue/CombineConsumeQueueStore.java Outdated

lizhimins reviewed Apr 18, 2025

View reviewed changes

Comment thread store/src/main/java/org/apache/rocketmq/store/queue/ConsumeQueueStore.java Outdated

lizhimins reviewed Apr 18, 2025

View reviewed changes

Comment thread store/src/main/java/org/apache/rocketmq/store/queue/ConsumeQueueStore.java

lizhimins reviewed Apr 18, 2025

View reviewed changes

Comment thread store/src/main/java/org/apache/rocketmq/store/queue/ConsumeQueueStore.java Outdated

lollipopjin reviewed Apr 21, 2025

View reviewed changes

Comment thread broker/src/main/java/org/apache/rocketmq/broker/processor/AdminBrokerProcessor.java Outdated