Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

In commit_txn request, we will add an optional boolean field writemanagedwriteManaged, which indicate the query modifies managed tables in the cache. If this is false, HMS won’t fetch writeid for the transaction from db. Note this is a performance optimization which does not impact the correctness.

hive_metastore.thriftOld API

New API

commitTxn(long txnid)

commitTxn(long txnid, boolean writemanagedwriteManaged)

RawStore

ObjectStore will use the additional validWriteIdList field for all read methods to compare with cached ValidWriteIdList

...

Optionally, HMS client (HiveMetaStoreClient) can set writemanaged flag writeManaged flag in commit transaction request (commitTxn) if this transaction modifies any managed table/partition. This will save a db fetch for HMS for readonly query or DDL/DML for external tables. If this set to true wrongly (eg, readonly query claim it modifies managed table), there will be a performance penalty processing commit message. If this set to false wrongly (eg, DDL on managed table claim it does not touch managed table), the entry in the cache will not mark available thus every read has to go to db. In both scenarios, there is no correctness issue.