Page History

...

When reading, we need to pass a writeid list so HMS can compare it with the cached version. For every commit, we need to tell HMS what is the writeid for the tables changed during the transaction, so HMS can correctly tag the entry with the writeid. We need to add ValidWriteIdList into metastore thrift API and RawStore interfaces, including all table/partition related read calls and commit transaction calls.

Hive_metastore.thriftthrift and HiveMetaStoreClient.java

Adding a serialized version of ValidWriteIdList to every read HMS API.

...

1. Many APIs are using a request structure rather than taking individual parameters. So need to add ValidWriteIdList to the request structure instead
2. Some APIs already take ValidWriteIdList to invalidate outdated transactional statistics. We don’t need to change the API signature, but will reuse the ValidWriteIdList to validate cached entries in CachedStore

...

HMS read API will remain backward compatible for external table. That is, new server can deal with old client. If the old client issue a create_table call, server side will receive the request of create_table with validWriteIdList=null, and will cache or retrieve the entry regardless(with eventual consistency model). For managed table, validWriteIdList will be required and HMS server will throw an exception if validWriteIdList=null.

In commit_txn request, we will add an optional boolean field writemanaged, which indicate the query modifies managed tables in the cache. If this is false, HMS won’t fetch writeid for the transaction from db. Note this is a performance optimization which does not impact the correctness.

RawStore

ObjectStore will use the additional validWriteIdList field for all read methods to compare with cached ValidWriteIdList

...

ValidTxnList txnIds = txnMgr.getValidTxns(); // get global transaction state
ValidTxnWriteIdList txnWriteIds = txnMgr.getValidWriteIds(txnTables, txnString); // map global transaction state to table specific write id

Optionally, HMS client (HiveMetaStoreClient) can set readonly writemanaged flag in commit transaction request (commitTxn) if this is a readonly transactiontransaction modifies any managed table/partition. This will save a db fetch for HMS for readonly query or DDL/DML for external tables. If this set to true wrongly (eg, readonly query claim it modifies managed table), there will be a performance penalty processing commit message. If this set to false wrongly (eg, DDL on managed table claim it does not touch managed table), the entry in the cache will not mark available thus every read has to go to db. In both scenarios, there is no correctness issue.