HIVE-29496: Subsequent transactional materialized view rebuild fails when source tables are iceberg tables#6434
Open
kasakrisz wants to merge 1 commit intoapache:masterfrom
Open
Conversation
kasakrisz
force-pushed
the
HIVE-29496-master-acid-mv-lock
branch
from
c31cc79 to
f5ab536
Compare
…when source tables are iceberg tables
kasakrisz
force-pushed
the
HIVE-29496-master-acid-mv-lock
branch
from
f5ab536 to
b347405
Compare
asf-ci-hive
added
tests pending
tests unstable
and removed
tests unstable
tests pending
labels
|
| create table ice_basetable (a int, b string) stored by iceberg; | ||
| insert into ice_basetable values (1, 'alfred'),(2, 'bob'),(2, 'bonnie'),(3, 'calvin'),(3, 'charlie'); | ||
|
|
||
| create materialized view mv_acid STORED AS ORC TBLPROPERTIES ('transactional'='true') as |
Contributor
There was a problem hiding this comment.
Thinking out loud: shouldn't we add a test case when the MV itself stored by iceberg?
Contributor
Author
There was a problem hiding this comment.
there are already tests like
https://github.com/apache/hive/blob/master/iceberg/iceberg-handler/src/test/queries/positive/mv_iceberg_orc2.q
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Move the lock acquisition for materialized view rebuilds to after the Hive operator plan generation.
Why are the changes needed?
Before plan generation, the current transaction ID defaults to 0, and this value is passed to the lock acquisition request. This prevents the subsequent rebuild of transactional MVs when the source tables are Iceberg.
The transaction ID is set during the Hive operator tree generation, specifically during FS operator creation. When source tables are ACID-compliant, the transaction ID is set before the rebuild plan is generated; therefore, this scenario remains unaffected."
Does this PR introduce any user-facing change?
No.
How was this patch tested?