Notification

Since version 0.2, HCatalog provides notifications for certain events happening in the system. This way applications such as Oozie can wait for those events and schedule the work that depends on them. The current version of HCatalog supports two kinds of events:

...

To receive notification that a new partition has been added, you need to follow these three steps.

1. To start receiving messages, create a connection to a message bus as shown here:

   No Format
   ConnectionFactory connFac = new ActiveMQConnectionFactory(amqurl); Connection conn = connFac.createConnection(); conn.start();

   
   

2. Subscribe to a topic you are interested in. When subscribing on a message bus, you need to subscribe to a particular topic to receive the messages that are being delivered on that topic.
   
   

   • The topic name corresponding to a particular table is stored in table properties and can be retrieved using the following piece of code:

     No Format
     HiveMetaStoreClient msc = new HiveMetaStoreClient(hiveConf); String topicName = msc.getTable("mydb", "myTbl").getParameters().get(HCatConstants.HCAT_MSGBUS_TOPIC_NAME);

     
     

   • Use the topic name to subscribe to a topic as follows:

     No Format
     Session session = conn.createSession(true, Session.SESSION_TRANSACTED); Destination hcatTopic = session.createTopic(topicName); MessageConsumer consumer = session.createConsumer(hcatTopic); consumer.setMessageListener(this);

     
     

3. To start receiving messages you need to implement the JMS interface MessageListener, which, in turn, will make you implement the method onMessage(Message msg). This method will be called whenever a new message arrives on the message bus. The message contains a partition object representing the corresponding partition, which can be retrieved as shown here:

   No Format
   @Override public void onMessage(Message msg) { // We are interested in only add_partition events on this table. // So, check message type first. if(msg.getStringProperty(HCatConstants.HCAT_EVENT).equals(HCatConstants.HCAT_ADD_PARTITION_EVENT)){ Object obj = (((ObjectMessage)msg).getObject()); } }

   
   

...

1. Repeat steps one and two from above to establish the connection to the notification system and to subscribe to the topic.
   

2. Receive the notification as shown in this example:

   No Format
   HiveMetaStoreClient msc = new HiveMetaStoreClient(conf); // Create a map, specifying partition key names and values Map<String,String> partMap = new HashMap<String, String>(); partMap.put("date","20110711"); partMap.put("country","*"); // Mark the partition as "done" msc.markPartitionForEvent("mydb", "mytbl", partMap, PartitionEventType.LOAD_DONE);

   
   

...