You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 9 Next »

HiveServer2 (HS2) is a service that enables clients to execute queries against Hive. HiveServer2 is success to HiveServer1 which has been deprecated. HS2 supports multi-client concurrency and authentication. It is designed to provide better support for open API clients like JDBC and ODBC.

HS2 is a single process running as a composite service, which includes the Thrift based Hive service (TCP or HTTP) and a Jetty web server for web UI. 

HS2 Architecture

The Thrift based Hive service is the core of HS2 and responsible for servicing the Hive queries (e.g. from Beeline). Thrift is a RPC framework for building cross-platform services. Its stack consists of 4 layers: Server, Transport, Protocol and Processor. You can find more details about the layers at: https://thrift.apache.org/docs/concepts .  

The usage of those layers from HS2 implementation is described as below.

Server

HS2 uses a TThreadPoolServer (from Thrift) for TCP mode, or a Jetty server for the HTTP mode. 

Regarding the TThreadPoolServer, it allocates one worker thread per TCP connection. Each thread is always associated with a connection even if the connection is idle. So there is a potential performance issue resulting from a large number of threads due to a large number of concurrent connections. In future we may think about switching to another server types for TCP mode, for example TThreadedSelectorServer. Here is a article about a performance comparison between different Thrift Java servers.  

Transport

HTTP mode is required when a proxy is needed between the client and server (for example, for load balancing or security reasons). That's why it's supported as well as TCP mode. You can specify the transport mode of the Thrift service through the hive config: "hive.server2.transport.mode".

Protocol

The Protocol implementation is responsible for serialization/deserialization. We are currently using TBinaryProtocol as our thrift protocol for serialization. In future we may think about other protocols such as TCompactProtocol based on more performance evaluation.

Processor

Process implementation is the application logic to handle requests. For example, ThriftCLIService.ExecuteStatement() method implements the logic to compile and execute a Hive query.

Dependencies of HS2

  • Metastore
    Metastore can be configured as embedded (in the same process as HS2) or a remote server (Thrift based service as well). HS2 talks to Metastore for metadata that is required for query compilation.  
  • Hadoop cluster
    HS2 prepares physical execution plans for various execution engines (MapReduce/Tez/Spark) and submits jobs to the Hadoop cluster for execution.

You can find a diagram of the interactions between HS2 and its dependencies here.

JDBC Client

JDBC driver is recommended for the client side to interact with HS2. Note that there are some use cases (e.g. Hadoop Hue) where Thrift client is used directly and JDBC is bypassed.

Here is a flow of API calls when making the first query:

  • The JDBC client (e.g. beeline) creates a HiveConnection by initiating a transport connection (e.g. TCP connection) followed by issuing a TOpenSessionReq to get the SessionHandle.
  • HiveStatement contains SessionHandle information and pass it to HS2 along with the query information.
  • HS2 receives the request and the driver (which is a CommandProcessor) is responsible for query parsing and compilation. HS2 kick off a background job to talk to Hadoop and then returns response to client. This is an asynchronous design of the ExecuteStatement API. The response contains OperationHandle information which client can use to poll the status of the query execution.  

Other resources

How to set up HS2: Setting Up HiveServer2

HS2 clients: HiveServer2 Clients

Cloudera blog: http://blog.cloudera.com/blog/2013/07/how-hiveserver2-brings-security-and-concurrency-to-apache-hive/

 

 

 

  • No labels