...

This is the approach we expect to take. One further modification this will require from the current export semantics, is that currently, export exports only 1 _metadata file per table, which contains the list of all the partitions inside it in the _metadata file itself. Instead, now, we propose to split that up so that the _metadata level at an object level will contain only metadata for that object. Thus, _metadata at a table level will contain only the table object, and the individual directories inside it will contain all the required partitions, and each of those dirs will have a partition level _metadata. 

Metastore notification API security

We want to secure DbNotificationListener related metastore APIs listed below by adding an authorization logic (other APIs not affected). These three APIs are mainly used by replication operations, so are allowed to be used by admin/superuser only:

1. get_next_notification
2. get_current_notificationEventId
3. get_notification_events_count

The related hive config parameter is "hive.metastore.event.db.notification.api.auth", which is set to true by default.

The auth mechanism works as below:

1. Skip auth in embedded metasore mode regardless of "hive.metastore.event.db.notification.api.auth" setting
   The reason is that we know the metastore calls are made from hive as opposed to other un-authorized processes that are running metastore client.
   
   
2. Enable auth in remote metastore mode if "hive.metastore.event.db.notification.api.auth" set to true
   The UGI of the remote metastore client is always set on metastore server. We retrieve this user info and check if this user has proxy privilege according to the proxy user settings. For example, the UGI is user "hive" and "hive" been configured to have the proxy privilege against a list of hosts. Then the auth will pass for the notification related calls from those hosts. If the user "foo" (e.g. an end user "foo" calling HS2 with doAs=true), then the auth will fail unless user "foo" is configured to have the proxy privilege.

Setup/Configuration

The following parameters need to be setup in source cluster -

...