Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.
Comment: Migrated to Confluence 5.3

– This document is a work in progress.

Overview

There are presently three ways to issue HCatalog DDL commands:

  1. Command line interface
  2. Upcoming REST APIs (upcoming)
  3. HiveMetaStore Client

Presently, java developers go through the Hive meta store (HMS) client interface to issue HCatalog DDl commands. Though the HMS client interface is public, it is not intended for public users. According to the hive user mailing list, the HMS client is not a public API and is subject to change in the future. So, it will be a good idea to have a java APIs in HCatalog which will provide a protect users from the changes made to the hive meta store client. Also, the under the covers either the Templeton Rest APIS or the hive metastore client can be used to provide end users with the required data.

...

Code Block
public abstract class HCatClient {

    /**
     * Creates an instance of HCatClient.
     *
     * @param conf An instance of configuration.
     * @return An instance of HCatClient.
     * @throws IOException
     */
    public static HCatClient create(Configuration conf) throws IOException{
      HCatClient client  return null= HCatUtil.getHCatClient(conf);
      }

  if(client  /**!= null){
     * Get all existing databases that match the given client.initialize(conf);
     * pattern. The matching occurs as per Java regular expressions
 }
        return *client;
    }

 * @param databasePattern
 abstract void initialize(Configuration conf) *throws HCatException;

    /**
     java* reGet pattern
all existing databases that match the given
  * @return list of* databasepattern. names
The matching occurs as per *Java @throwsregular HCatExceptionexpressions
     */
    public abstract* List<String> listDatabasesByPattern(String pattern) throws HCatException;

@param databasePattern
     /**
     * Gets the database.
   java re *pattern
     * @param dbName The name@return list of the database.
     * @return An instance of HCatDatabaseInfo. names
     * @throws HCatException
     */
    public abstract HCatDatabaseList<String> getDatabaselistDatabaseNamesByPattern(String dbNamepattern) throws HCatException;

    /**
     * CreatesGets the database.
     *
     * @param dbInfo dbName The name of the database.
     * @return An instance of HCatCreateDBDescHCatDatabaseInfo.
     * @throws HCatException
     */
    public abstract voidHCatDatabase createDatabasegetDatabase(HCatCreateDBDescString dbInfo)
           dbName) throws HCatException;

    /**
     * DropsCreates athe database.
     *
     * @param dbNamedbInfo TheAn nameinstance of the database to deleteHCatCreateDBDesc.
     * @throws HCatException
 @param ifExists Hive returns an*/
 error if the databasepublic specifiedabstract doesvoid not exist,createDatabase(HCatCreateDBDesc dbInfo)
     *       throws HCatException;

    /**
     * Drops a unlessdatabase.
 ifExists is set to true.*
     * @param modedbName ThisThe isname setof tothe eitherdatabase "restrict" or "cascade". Restrict willto delete.
     * @param ifExists Hive returns an error if the database specified does not removeexist,
 the schema if all the* tables are empty. Cascade removes
     *       unless ifExists is set to true.
 everything including data and definitions.
* @param mode This is *set @throwsto HCatException
either "restrict" or "cascade". Restrict */will
     * public abstract void dropDatabase(String dbName, boolean ifExists, String mode) throws HCatException;

    /**
   remove the *schema Returnsif all existingthe tables fromare theempty. specified database which match the givenCascade removes
     * pattern. The matching occurs as per Java regular expressions.
    everything *including @paramdata dbName
     * @param tablePattern
     * @return list of table namesand definitions.
     * @throws HCatException
     */
    public abstract List<String>void listTablesByPatterndropDatabase(String dbName, boolean ifExists, String tablePattern)
           mode) throws HCatException;

    /**
     * Returns Getsall the table.
     *existing tables from the specified database which match the given
     * pattern. The @parammatching dbNameoccurs Theas nameper ofJava theregular databaseexpressions.
     * @param dbName
  tableName The name of* the@param table.tablePattern
     * @return Anlist instanceof oftable HCatTableInfo.names
     * @throws HCatException
     */
    public abstract HCatTableList<String> getTablelistTableNamesByPattern(String dbName, String tableNametablePattern)
            throws HCatException;

    /**
     * CreatesGets the table.
     *
     * @param createTableDescdbName AnThe instancename of HCatCreateTableDescthe classdatabase.
     * @param @throwstableName HCatExceptionThe thename hof catthe exceptiontable.
     */
 @return An instance public abstract void createTable(HCatCreateTableDesc createTableDesc)of HCatTableInfo.
            throws* @throws HCatException;

     */**
    public *abstract CreatesHCatTable the table like an existing table.getTable(String dbName, String tableName)
     *
     * @param dbName The name of the database.throws HCatException;

    /**
     * @paramCreates existingTblNamethe Thetable.
 name of the existing table.*
     * @param newTableNamecreateTableDesc TheAn nameinstance of theHCatCreateTableDesc new tableclass.
     * @param@throws ifExistsHCatException the h ifcat existsexception
     */
 @param isExternal Set to "true", if table has be created at a different
public abstract void createTable(HCatCreateTableDesc createTableDesc)
          *  throws HCatException;

    /**
     * Creates the table like an  location other than defaultexisting table.
     *
     * @param locationdbName The locationname forof the tabledatabase.
     * @param @throwsexistingTblName HCatException
The name of the existing */table.
    public abstract* void@param createTableLike(String dbName, String existingTblName,
   newTableName The name of the new table.
     * @param ifNotExists If String newTableNametrue, booleanthen ifExists,error booleanrelated isExternal,
to already table existing is skipped.
     * @param StringisExternal location)Set throws HCatException;

    /**
     * Drop table.to "true", if table has be created at a different
     *
     * @param dbName The name of the database.
     * @param tableNamelocation Theother name of the tablethan default.
     * @param location ifExistsThe Hivelocation returnsfor anthe errortable.
 if the database specified does* not@throws exist,HCatException
     */
    public abstract void createTableLike(String dbName, String existingTblName,
       unless ifExists is set to true.
String newTableName, boolean ifNotExists,  * @throws HCatException
  boolean isExternal,
   */
    public abstract void dropTable(String dbName, String tableName,
            boolean ifExistslocation) throws HCatException;

    /**
     * Renames aDrop table.
     *
     * @param dbName The name of the database.
     * @param oldNametableName The name of the table to be renamed.
     * @param ifExists newNameHive Thereturns newan nameerror ofif the table.
   database specified does not exist,
  * @throws HCatException
 *    */
    public abstract void renameTable(String dbName, String oldName, String newName) throws HCatException;

    /**unless ifExists is set to true.
     * Gets all the partitions.@throws HCatException
     */
    public *abstract @paramvoid dropTable(String dbName, TheString nametableName,
 of the database.
     * @param tblName The nameboolean ofifExists) the table.
throws HCatException;

    /**
  * @return A list* ofRenames partitiona namestable.
     *
 @throws HCatException the h cat* exception
@param dbName The name  */
of the database.
     public* abstract@param List<HCatPartition>oldName getPartitions(String dbName, String tblName)
    The name of the table to be renamed.
     * @param newName throws HCatException;

    /**The new name of the table.
     * Gets the partition.@throws HCatException
     */
    public abstract *void @param dbName The database name.renameTable(String dbName, String oldName, String newName) throws HCatException;

    /**
     * @paramGets tableNameall Thethe table namepartitions.
     *
  @param partitionName The partition* name,@param CommadbName separatedThe listname of col_name='value' the database.
     * @return An instance of HCatPartitionInfo@param tblName The name of the table.
     * @return A list of partition names.
     * @throws HCatException the h cat exception
     */
    public abstract HCatPartitionList<HCatPartition> getPartitiongetPartitions(String dbName, String tableName,tblName)
            String partitionName) throws HCatException;

    /**
     * AddsGets the partition.
     *
     * @param partInfodbName AnThe instancedatabase of HCatAddPartitionDescname.
     * @throws@param HCatExceptiontableName theThe h cat exceptiontable name.
     */
 @param partitionName The publicpartition abstractname, voidComma addPartition(HCatAddPartitionDesc partInfo) throws HCatException;

separated list of col_name='value'.
     /*** @return An instance of HCatPartitionInfo.
     * Drops@throws partition.HCatException
     */
    public abstract *HCatPartition @paramgetPartition(String dbName The database name.
, String tableName,
       * @param tableName The table name.
String partitionName) throws HCatException;

  * @param partitionName/**
 The partition name, Comma separated* listAdds of col_name='value'the partition.
     *
 @param ifExists Hive returns an* error@param ifpartInfo theAn partitioninstance specified does not exist, unless ifExists is set to trueof HCatAddPartitionDesc.
     * @throws HCatException the h cat exception
     */
    public abstract void dropPartitionaddPartition(StringHCatAddPartitionDesc dbName,partInfo) String tableName,throws HCatException;

    /**
     *   String partitionName, boolean ifExists) throws HCatException;

    /**
     * List partitions by filterDrops partition.
     *
     * @param dbName The database name.
     * @param tblNametableName The table name.
     * @param filterpartitionName The filterpartition stringname,
 Comma separated list  *of col_name='value'.
    for example "part1 = \"p1_abc\" and part2 <= "\p2_test\"". Filtering can
     *    be done only on string partition keys. * @param ifExists Hive returns an error if the partition specified does not exist, unless ifExists is set to true.
     * @throws HCatException
     */
    public abstract void dropPartition(String dbName, String tableName,
            String partitionName, boolean ifExists) throws HCatException;

    /**
     * List partitions by filter.
     *
     * @param dbName The database name.
     * @param tblName The table name.
     * @param filter The filter string,
     *    for example "part1 = \"p1_abc\" and part2 <= "\p2_test\"". Filtering can
     *    be done only on string partition keys.
     * @return list of partitions
     * @throws HCatException the h cat exception
     */
    public abstract List<HCatPartition> listPartitionsByFilter(String dbName, String tblName,
            String filter) throws HCatException;

    /**
     * Mark partition for event.
     *
     * @param dbName The database name.
     * @param tblName The table name.
     * @param partKVs the part k vs
     * @param @returneventType listthe ofevent partitionstype
     * @throws HCatException the h cat exception
     */
    public abstract void markPartitionForEvent(String dbName, String tblName,
     */
    public abstract List<HCatPartition> listPartitionsByFilter(String dbNameMap<String, StringString> tblNamepartKVs,
 PartitionEventType eventType)
          String filter) throws HCatException;

    /**
     * Checks if Markis partition marked for event.
     *
     * @param dbName Thethe databasedb name.
     * @param tblName Thethe tabletbl name.
     * @param partKVs the part k vs
     * @param eventType the event type
     * @return true, if *is @parampartition eventTypemarked thefor event type
     * @throws HCatException the h cat exception
     */
    public abstract voidboolean markPartitionForEventisPartitionMarkedForEvent(String dbName, String tblName,
            Map<String, String> partKVs, PartitionEventType eventType)
            throws HCatException;

    /**
     * ChecksGets ifthe is partition marked for event.
     *
     * @param dbName the db name
     * @param tblName the tbl namedelegation token.
     *
     * @param partKVsowner the part k vsowner
     * @param eventTyperenewerKerberosPrincipalName the renewer kerberos eventprincipal typename
     * @return true,the if is partition marked for eventdelegation token
     * @throws HCatException the h cat exception
     */
    public abstract boolean isPartitionMarkedForEvent(String dbName, String tblName,
         public abstract String Map<String, String> partKVsgetDelegationToken(String owner, PartitionEventTypeString eventTyperenewerKerberosPrincipalName) throws
            throws HCatException;

    /**
     * Gets theRenew delegation token.
     *
     * @param renewerKerberosPrincipalNametokenStrForm the renewertoken kerberosstr principal nameform
     * @return the delegation tokenlong
     * @throws HCatException the h cat exception
     */
    public abstract Stringlong getDelegationToken(String renewerKerberosPrincipalName) throws
       renewDelegationToken(String tokenStrForm) throws HCatException;

    /**
     * RenewCancel delegation token.
     *
     * @param tokenStrForm the token str form
     * @return the long
     * @throws HCatException the h cat exception
     */
    public abstract longvoid renewDelegationTokencancelDelegationToken(String tokenStrForm) throws HCatException;

    /**
     * Close Cancelthe delegationhcatalog token.
     *client.
     * @param tokenStrForm the token str form
     * @throws HCatException the h cat exception
     */
    public abstract void cancelDelegationTokenclose(String tokenStrForm) throws HCatException;

}    throws HCatException;
HCatCreateTableDesc

This class is a sub class of HCatCommandDesc and will be used by the users to create descriptor and validate it for the "create table" command.
Image Removed Image Added

HCatCreateDBDesc

This class is a sub class of HCatCommandDesc and will be used by the users to create descriptors and validate it for the "create database" command.

Image Removed!createdb.png|

HCatAddPartitionDesc

This class is a sub class of HCatCommandDesc and will be used by the users to create descriptors and validate it for the "add partition" command.



HCatTable

This class encapsulates the table information returned the HCatClient implementation class and provides a uniform view to the user.

...

Code Block
 Configuration config = new Configuration();
 config.add("hive-site.xml");
 HCatClient client = HCatClient.create(config);
 ArrayList<HCatFieldSchema> cols = new ArrayList<HCatFieldSchema>();
 cols.add(new HCatFieldSchema("col1id", Type.INT, "comment1id columns"));
 cols.add(new HCatFieldSchema("col2value", Type.STRING, "comment2id columns"));
HCatCreateTableDesc desctableDesc = new HCatCreateTableDesc.Buildercreate(db, "defaulttesttable","demo_table" cols).setFileFormatfileFormat("rcfile").setCols(cols).build();

 boolean success = client.createTable(desctableDesc);

Discussion Topics