Notification
Table of Contents |
---|
Since version 0.2, HCatalog provides notifications for certain events happening in the system. This way applications such as Oozie can wait for those events and schedule the work that depends on them. The current version of HCatalog supports two kinds of events:
...
To receive notification that a new partition has been added, you need to follow these three steps.
To start receiving messages, create a connection to a message bus as shown here:
No Format ConnectionFactory connFac = new ActiveMQConnectionFactory(amqurl); Connection conn = connFac.createConnection(); conn.start();
- Subscribe to a topic you are interested in. When subscribing on a message bus, you need to subscribe to a particular topic to receive the messages that are being delivered on that topic.
The topic name corresponding to a particular table is stored in table properties and can be retrieved using the following piece of code:
No Format HiveMetaStoreClient msc = new HiveMetaStoreClient(hiveConf); String topicName = msc.getTable("mydb", "myTbl").getParameters().get(HCatConstants.HCAT_MSGBUS_TOPIC_NAME);
Use the topic name to subscribe to a topic as follows:
No Format Session session = conn.createSession(true, Session.SESSION_TRANSACTED); Destination hcatTopic = session.createTopic(topicName); MessageConsumer consumer = session.createConsumer(hcatTopic); consumer.setMessageListener(this);
To start receiving messages you need to implement the JMS interface
MessageListener
, which, in turn, will make you implement the methodonMessage(Message msg)
. This method will be called whenever a new message arrives on the message bus. The message contains a partition object representing the corresponding partition, which can be retrieved as shown here:No Format @Override public void onMessage(Message msg) { // We are interested in only add_partition events on this table. // So, check message type first. if(msg.getStringProperty(HCatConstants.HCAT_EVENT).equals(HCatConstants.HCAT_ADD_PARTITION_EVENT)){ Object obj = (((ObjectMessage)msg).getObject()); } }
...
- Repeat steps one and two from above to establish the connection to the notification system and to subscribe to the topic.
Receive the notification as shown in this example:
No Format HiveMetaStoreClient msc = new HiveMetaStoreClient(conf); // Create a map, specifying partition key names and values Map<String,String> partMap = new HashMap<String, String>(); partMap.put("date","20110711"); partMap.put("country","*"); // Mark the partition as "done" msc.markPartitionForEvent("mydb", "mytbl", partMap, PartitionEventType.LOAD_DONE);
...
For the server to start with support for notifications, the following must be in the classpath:
..... (a) activemq
jar
..... (b) jndi.properties
file with properties suitably configured for notifications
...
- The HCatalog server start script is $YOUR_HCAT_SERVER
/share/hcatalog/scripts/hcat_server_start.sh
. - This script expects classpath to be set by the AUX_CLASSPATH environment variable.
- Therefore set AUX_CLASSPATH to satisfy (a) and (b) above.
- The
jndi.properties
file is located at $YOUR_HCAT_SERVER/etc/hcatalog/jndi.properties
. - You need to uncomment and set the following properties in the
jndi.properties
file:java.naming.factory.initial = org.apache.activemq.jndi.ActiveMQInitialContextFactory
java.naming.provider.url = tcp://localhost:61616
.. (This is the ActiveMQ URL in your setup.)
...
If tables are created while the server is configured for notifications, a default topic name is automatically set as a table property. To use notifications with tables created previously (either in other HCatalog installations or prior to enabling notifications in the current installation) you will have to manually set a topic name. For example:
..... $YOUR_HCAT_CLIENT_HOME/bin/hcat -e "ALTER TABLE access SET hcat.msgbus.topic.name=$TOPIC_NAME"
...
Panel | ||||||
---|---|---|---|---|---|---|
| ||||||
Previous: Dynamic Partitioning General: HCatalog Manual – WebHCat (Templeton) Manual – Hive Home |