Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.
Comment: Migrated to Confluence 5.3

– This document is a work in progress.

Overview

There are presently three ways to issue HCatalog DDL commands:

  1. Command line interface
  2. Templeton REST APIs (upcoming)
  3. HiveMetaStore Client

Presently, java developers go through the Hive meta store (HMS) client interface to issue HCatalog DDl commands. Though the HMS client interface is public, it is not
intended for public users. According to the hive user mailing list, the HMS client is not a public API and is subject to change in the future. So, it will be a good idea
to have a java APIs in HCatalog which will provide a protect users from the changes made to the hive meta store client. Also, the under the covers either the
Templeton Rest APIS or the hive metastore client can be used to provide end users with the required data.

Design

Image RemovedImage Added

New Classes

HCatClient

The HCatClient is an abstract class containing all the APIs permitted HCatalog DDL commands. The implementation class will be provided as a configuration property, which will be used by the
"create" method. In this way, the implementation details will be masked to the users.

Code Block
/**
 * The abstract class HCatClient containing APIs for HCatalog DDL commands.
 */
public abstract class HCatClient {

    /**
     * Creates an instance of HCatClient.
     *
     * @param conf An instance of configuration.
     * @return An instance of HCatClient.
     * @throws IOException
     */
    public static HCatClient create(Configuration conf){
 throws IOException{
      //HCatClient Obtainclient details of implementation class and return an instance.     = HCatUtil.getHCatClient(conf);
        if(client != null){
    }

    /**
     * Gets the database like.
client.initialize(conf);
          *}
     * @param regex Thereturn regularclient;
 expression. Providing "*" would retrieve all the names
     * }

    abstract void initialize(Configuration conf) throws HCatException;

    /**
     * Get all existing databases that ofmatch the databases.given
     * @returnpattern. The listmatching occurs ofas allper theJava databaseregular names.expressions
     *
     * @param databasePattern
     *          java re pattern
     * @return list of database names
     * @throws HCatException
     */
    public abstract List<String> getDatabaseLikelistDatabaseNamesByPattern(String regexpattern) throws HCatException;

    /**
     * Gets the database.
     *
     * @param dbName The name of the database.
     * @return An instance of HCatDatabaseInfo.
     * @throws HCatException
     */
    public abstract HCatDatabase getDatabase(String dbName) throws HCatException;

    /**
     * Creates the database.
     *
     * @param dbInfo An instance of HCatCreateDBDesc.
     * @return@throws true,HCatException
 if successful
     * @throws HCatException
     */
    public abstract booleanvoid createDatabase(HCatCreateDBDesc dbInfo)
            throws HCatException;

    /**
     * DeletesDrops a database.
     *
     * @param dbName The name of the database to delete.
     * @param ifExists Hive returns an error if the database specified does not exist,
     *                 unless ifExists is set to true.
     * @param mode This is set to either "restrict" or "cascade". Restrict will
     *             remove the schema if all the tables are empty. Cascade removes
     *             everything including data and definitions.
     * @param@throws userGroupHCatException
 The user group to use*/
    public abstract * @param permissions The permissions string to use. The format is "rwxrw-r-x".void dropDatabase(String dbName, boolean ifExists, String mode) throws HCatException;

    /**
     * @return true, if successful
     * @throws HCatExceptionReturns all existing tables from the specified database which match the given
     */
 pattern. The matching publicoccurs abstractas booleanper deleteDatabase(String dbName, boolean ifExists, String mode,
    Java regular expressions.
     * @param dbName
     * @param tablePattern
 String userGroup, String permissions) throws HCatException;

    /*** @return list of table names
     * Gets@throws theHCatException
 tables like a pattern specified.*/
    public *
abstract List<String> listTableNamesByPattern(String dbName, String *tablePattern)
 @param dbName The name of the database.
     * @param regex The regular expression. Providing "*" would retrieve all the namesthrows HCatException;

    /**
     * Gets the table.
     *
     * @param dbName The name     of  the tabledatabase.
     * @param @returntableName AThe listname of allthe table names matching the specified pattern.
     * @return An instance of HCatTableInfo.
     * @throws HCatException
     */
    public abstract List<String>HCatTable getTablesLikegetTable(String dbName, String regextableName)
            throws HCatException;

    /**
     * GetsCreates the table.
     *
     * @param dbNamecreateTableDesc TheAn nameinstance of theHCatCreateTableDesc databaseclass.
     * @param tableName The name of the table.
     * @return An instance of HCatTableInfo.@throws HCatException the h cat exception
     * @throws HCatException
     */
    public abstract HCatTablevoid getTablecreateTable(String dbName, String tableNameHCatCreateTableDesc createTableDesc)
            throws HCatException;

    /**
     * Creates the table like an existing table.
     *
     * @param createTableDescdbName AnThe instancename of HCatCreateTableDescthe classdatabase.
     * @return true, if successful @param existingTblName The name of the existing table.
     * @throws HCatException@param newTableName The name of the h cat exceptionnew table.
     */
 @param ifNotExists If publictrue, abstractthen booleanerror createTable(HCatCreateTableDesc createTableDesc)related to already table existing is skipped.
     * @param isExternal Set to "true", if throws HCatException;

    /**table has be created at a different
     * Creates the table like an existing table.
     *
     * @param dbNamelocation Theother name of the databasethan default.
     * @param existingTblNamelocation The namelocation offor the existing table.
     * @param@throws newTableNameHCatException
 The name of the new table. */
    public *abstract @paramvoid ifExists the if existscreateTableLike(String dbName, String existingTblName,
     * @param isExternal Set to "true", if tableString hasnewTableName, beboolean createdifNotExists, at a differentboolean isExternal,
     *       String location) throws HCatException;

    /**
     * Drop table.
  location other than default.*
     * @param locationdbName The locationname forof the tabledatabase.
     * @return true, if successful@param tableName The name of the table.
     * @throws@param HCatException
ifExists Hive returns an error */
if the database specified publicdoes abstract boolean createTableLike(String dbName, String existingTblName,
not exist,
     *         String newTableName, boolean ifExists, boolean isExternal,
   unless ifExists is set to true.
    String location)* throws@throws HCatException;

     */**
    public abstract * Delete a table.void dropTable(String dbName, String tableName,
     *
     * @param dbNameboolean TheifExists) name of the database.throws HCatException;

    /**
 * @param tableName The name* ofRenames thea table.
     *
 @param ifExists Hive returns an* error@param ifdbName theThe databasename specifiedof does not exist,the database.
     * @param oldName The name of the table to be renamed.
     * @param unlessnewName The ifExistsnew isname setof tothe truetable.
     * @param@throws userGroupHCatException
 The user group to use.*/
    public abstract * @param permissions The permissions string to use. The format is "rwxrw-r-x".void renameTable(String dbName, String oldName, String newName) throws HCatException;

    /**
     * @returnGets true,all ifthe successfulpartitions.
     * @throws HCatException
     */
 @param dbName The publicname abstractof boolean deleteTable(String dbName, String tableName,
            boolean ifExists, String userGroup, String permissionsthe database.
     * @param tblName The name of the table.
     * @return A list of partition names.
     * @throws HCatException the h cat exception
     */
    public abstract List<HCatPartition> getPartitions(String dbName, String tblName)
            throws HCatException;

    /**
     * RenamesGets athe tablepartition.
     *
     * @param dbName The database name.
 of the database.
     * @param oldNametableName The name of the table to be renamedname.
     * @param newNamepartitionName The newpartition name, of the tableComma separated list of col_name='value'.
     * @param@return userGroupAn Theinstance user group to useof HCatPartitionInfo.
     * @param@throws permissionsHCatException
 The permissions string to use. The format is "rwxrw-r-x". */
    public abstract HCatPartition getPartition(String dbName, String tableName,
     * @return true, if successful
   String partitionName) *throws @throws HCatException;

     /**/
    public abstract* booleanAdds renameTable(String dbName, String oldName, String newName,the partition.
     *
     * @param partInfo An instance of HCatAddPartitionDesc.
     * @throws HCatException the h cat exception
  String userGroup, String permissions   */
    public abstract void addPartition(HCatAddPartitionDesc partInfo) throws HCatException;

    /**
     * Gets all the partitionsDrops partition.
     *
     * @param dbName The database name of the database.
     * @param tblNametableName The table name.
   of the table.
* @param partitionName The partition *name, @returnComma Aseparated list of partition namescol_name='value'.
     * @throws@param HCatExceptionifExists theHive hreturns catan exception
error if the partition specified */
does not exist, unless ifExists is set to true.
     * @throws HCatException
     */
    public abstract List<HCatPartition>void getPartitionsdropPartition(String dbName, String tblName)tableName,
            String partitionName, boolean ifExists) throws HCatException;

    /**
     * GetsList partitions theby partitionfilter.
     *
     * @param dbName The database name.
     * @param tableNametblName The table name.
     * @param partitionNamefilter The partitionfilter namestring,
 Comma   separated list* of col_name='value'.
  for example "part1 * @return An instance of HCatPartitionInfo.= \"p1_abc\" and part2 <= "\p2_test\"". Filtering can
     * @throws HCatException
     */   be done only on string partition keys.
     * @return list of partitions
     * @throws HCatException the h cat exception
     */
    public abstract HCatPartitionList<HCatPartition> getPartitionlistPartitionsByFilter(String dbName, String tableNametblName,
            String partitionNamefilter) throws HCatException;

    /**
     * AddsMark partition thefor partitionevent.
     *
     * @param partInfodbName AnThe instance of HCatAddPartitionDesc.database name.
     * @return true, if successful@param tblName The table name.
     * @throws@param HCatExceptionpartKVs the hpart catk exceptionvs
     */
 @param eventType the publicevent abstracttype
 boolean addPartition(HCatAddPartitionDesc partInfo) throws HCatException;

    /**
 @throws HCatException the h * Deletes partition.cat exception
     */
    public * @paramabstract void markPartitionForEvent(String dbName, TheString database name.
tblName,
      * @param tableName The table name.
 Map<String, String> partKVs,  * @param partitionName The partition name, Comma separated list of col_name='value'.PartitionEventType eventType)
            throws HCatException;

     /**
 @param ifExists Hive returns an* errorChecks if theis partition specifiedmarked does not exist, unless ifExists is set to true.for event.
     *
     * @param userGroupdbName Thethe user group to use.db name
     * @param permissionstblName Thethe permissionstbl stringname
 to use. The format is "rwxrw-r-x".
     * @param partKVs the part k vs
     * @param eventType the event type
     * @return true, if successful is partition marked for event
     * @throws HCatException the h cat exception
     */
    public abstract boolean deletePartitionisPartitionMarkedForEvent(String dbName, String tableNametblName,
            String partitionNameMap<String, booleanString> ifExistspartKVs, StringPartitionEventType userGroup,eventType)
            String permissions) throws HCatExceptionthrows HCatException;

    /**
     * ListGets partitionsthe bydelegation filtertoken.
     *
     * @param dbNameowner The database name.the owner
     * @param tblName The tablerenewerKerberosPrincipalName the renewer kerberos principal name.
     * @param@return filterthe The filter string,delegation token
     * @throws HCatException the forh examplecat "part1exception
 = \"p1_abc\" and part2 <= "\p2_test\"". Filtering can
     *    be done only on string partition keys. */
    public abstract String getDelegationToken(String owner, String renewerKerberosPrincipalName) throws
        HCatException;

     * @return list of partitions/**
     * @throwsRenew HCatException the h cat exceptiondelegation token.
     */
    public abstract* @param List<HCatPartition>tokenStrForm listPartitionsByFilter(String dbName, String tblName,the token str form
     * @return the long
    String filter)* throws@throws HCatException;

 the h cat exception
     */**
    public abstract *long Mark partition for event.renewDelegationToken(String tokenStrForm) throws HCatException;

     /**
     * @paramCancel dbName The database namedelegation token.
     * @param tblName The table name.
     * @param partKVstokenStrForm the parttoken kstr vsform
     * @param eventType the event type
     * @throws HCatException the h cat exception
     */
    public abstract void markPartitionForEventcancelDelegationToken(String dbName,tokenStrForm) String tblName,throws HCatException;

    /**
     * Close the Map<String, String> partKVs, PartitionEventType eventType)
       hcatalog client.
     throws HCatException;

    /**
     * Checks@throws ifHCatException isthe partitionh marked for event.cat exception
     */
    public *abstract @paramvoid dbNameclose() the db name
     * @param tblName the tbl name
     * @param partKVs the part k vs
     * @param eventType the event type
     * @return true, if is partition marked for event
     * @throws HCatException the h cat exception
     */
    public abstract boolean isPartitionMarkedForEvent(String dbName, String tblName,
            Map<String, String> partKVs, PartitionEventType eventType)
            throws HCatException;

    /**
     * Gets the delegation token.
     *
     * @param owner the owner
     * @param renewerKerberosPrincipalName the renewer kerberos principal name
     * @return the delegation token
     * @throws HCatException the h cat exception
     */
    public abstract String getDelegationToken(String owner, String renewerKerberosPrincipalName) throws
        HCatException;

    /**
     * Renew delegation token.
     *
     * @param tokenStrForm the token str form
     * @return the long
     * @throws HCatException the h cat exception
     */
    public abstract long renewDelegationToken(String tokenStrForm) throws HCatException;

    /**
     * Cancel delegation token.
     *
     * @param tokenStrForm the token str form
     * @throws HCatException the h cat exception
     */
    public abstract void cancelDelegationToken(String tokenStrForm) throws HCatException;

}
HCatCommandDesc

This is an abstract class that helps in validating user input, building valid command descriptors and queries.

Code Block

/**
 * The Class HCatCommandDesc contains methods which help in validating,
 * building command descriptors and queries.
 */
public abstract class HCatCommandDesc{

    public abstract void validateCommandDesc() throws HCatException;
    abstract String buildQuery() throws HCatException;
    abstract boolean isValidationComplete();

}throws HCatException;
HCatCreateTableDesc

This class is a sub class of HCatCommandDesc and will be used by the users to create descriptor and validate it for the "create table" command.
Image Removed Image Added

HCatCreateDBDesc

This class is a sub class of HCatCommandDesc and will be used by the users to create descriptors and validate it for the "create database" command.

Image Removed!createdb.png|

HCatAddPartitionDesc

This class is a sub class of HCatCommandDesc and will be used by the users to create descriptos descriptors and validate it for the "add partition" command.

Image Removed Image Added

HCatTable

This class encapsulates the table information returned the HCatClient implementation class and provides a uniform view to the user.

Image RemovedImage Added

HCatDatabase

This class encapsulates the database information returned the HCatClient implementation class and provides a uniform view to the user.

Image RemovedImage Added

HCatPartition

This class encapsulates the partition information returned the HCatClient implementation class and provides a uniform view to the user.

Image RemovedImage Added

Usage

Code Block
 Configuration config = new Configuration();
 config.add("hive-site.xml");
 HCatClient client = HCatClient.create(config);

 HCatCreateTableDesc desc = new HCatCreateTableDesc();
 desc.setTableName("demo_table");
 desc.setDatabaseName("db1");
 desc.setFileFormat("rcfile".create(config);
 ArrayList<HCatFieldSchema> cols = new ArrayList<HCatFieldSchema>();
 cols.add(new HCatFieldSchema("col1id", Type.INT, "comment1id columns"));
 cols.add(new HCatFieldSchema("col2value", Type.STRING, "comment2id columns"));
 desc.setCols(cols);

 //Validate
 desc.validateCommandDescHCatCreateTableDesc tableDesc = HCatCreateTableDesc.create(db, "testtable", cols).fileFormat("rcfile").build();
 boolean success = client.createTable(desctableDesc);

Discussion Topics