THIS IS A TEST INSTANCE. ALL YOUR CHANGES WILL BE LOST!!!!
| Table of Contents |
|---|
Live Long and Process (LLAP) functionality was added in Hive 2.0 (HIVE-7926 and associated tasks). HIVE-9850 links documentation, features, and issues for this enhancement.
...
- These nodes are stateless. Any request to an LLAP node contains the data location and metadata. It processes local and remote locations; locality is the caller’s responsibility (YARN).
- Recovery/resiliency. Failure and recovery is simplified because any data node can still be used to process any fragment of the input data. The Tez AM can thus simply rerun failed fragments on the cluster.
- Communication between nodes. LLAP nodes are able to share data (e.g., fetching partitions, broadcasting fragments). This is realized with the same mechanisms used in Tez.
Execution Engine
LLAP will work works within existing, process-based Hive execution to preserve the scalability and versatility of Hive. It will does not replace the existing execution model but enhance rather enhances it.
- The daemons are optional. Hive will continue to can work without them and will also be is able to bypass them even if they are deployed and operational. Feature parity with regard to language features will be is maintained.
- External orchestration and execution engines. LLAP is not an execution engine (like MR MapReduce or Tez). Overall execution will be is scheduled and monitored by an existing Hive execution engine (such as Tez; ) transparently over both LLAP nodes, as well as regular containers. Obviously, LLAP level of support will depend depends on each individual execution engine (starting with Tez). MapReduce support is not planned, but other engines may be added later. Other frameworks like Pig will also have the choice of using LLAP daemons.
- Partial execution. The result of the work performed by an LLAP daemon can either form part of the result of a Hive query, or be passed on to external Hive tasks, depending on the query.
. YARN will remain remains responsible for the management and allocation of resources. The YARN container delegation model will be is used for users to allow the transfer of allocated resources to LLAP. To avoid the limitations of JVM memory settings, we will keep cached data is kept off-heap, as well as large buffers for processing (e.g., group by, joins), off-heap. This way, the daemon can use a small amount of memory, and additional resources (i.e., CPU and memory) will be assigned based on workload.Anchor Resource management Resource management
...
For partial execution as described above, LLAP nodes will execute “query fragments” such as filters, projections, data transformations, partial aggregates, sorting, bucketing, hash joins/semi-joins, etc. Only Hive code and blessed UDFs will be are accepted in LLAP. No code will be is localized and executed on the fly. This is done for stability and security reasons.
- Parallel execution. The An LLAP node will allow allows parallel execution for multiple query fragments from different queries and sessions.
- Interface. Users can access LLAP nodes directly via client API. They will be are able to specify relational transformations and read data via record-oriented streams.
I/O
The daemon will off-load loads I/O and transformation from compressed format to separate threads. The data will be is passed on to execution as it becomes ready, so the previous batches can be processed while the next ones are being prepared. The data will be is passed to execution in a simple RLE-encoded columnar format that is ready for vectorized processing; this will is also be the caching format, and intends with the intent to minimize copying between I/O, cache, and execution.
- Multiple file formats. I/O and caching depend on some knowledge of the underlying file format (especially if it is to be done efficiently). Therefore, similar to Vectorization work, different file formats will be supported through plugins specific to each format (starting with ORC). Additionally, a generic, less-efficient plugin may be added that supports any Hive input format. The plugins have to maintain metadata and transform the raw data to column chunks.
- Predicates and bloom filters. SARGs and bloom filters will be are pushed down to storage layer, if they are supported.
Caching
The daemon will cache caches metadata for input files, as well as the data. The metadata and index information can be cached even for data that is not currently cached. Metadata will be is stored in process in Java objects; cached data will be is stored in the format described in the I/O section, and kept off-heap (see Resource management).
- Eviction policy. The eviction policy will be is tuned for analytical workloads with frequent (partial) table-scans. Initially, a simple policy like LRFU will be is used. The policy will be is pluggable.
- Caching granularity. Column-chunks will be are the unit of data in the cache. This achieves a compromise between low-overhead processing and storage efficiency. The granularity of the chunks depends on the particular file format and execution engine (Vectorized Row Batch size, ORC stripe, etc.).
Workload Management
YARN will be is used to obtain resources for different workloads. Once resources (CPU, memory, etc.) have been obtained from YARN for a specific workload, the execution engine can choose to delegate these resources to LLAP, or to launch Hive executors in separate processes. Resource enforcement via YARN has the advantage of ensuring that nodes do not get overloaded, either by LLAP or by other containers. The daemons themselves will be is under YARN’s control.
ACID Support
LLAP will be is aware of transactions. The merging of delta files to produce a certain state of the tables will be is performed before the data is placed in cache.
Multiple versions are possible and the request will specify specifies which version is to be used. This has the benefit of doing the merge async asynchronously and only once for cached data, thus avoiding the hit on the operator pipeline.
...
LLAP servers are a natural place to enforce access control at a more fine-grained level than “per file”. Since the daemons know which columns and records are processed, policies on these objects can be enforced. This is not intended to replace the current mechanisms, but rather to enhance and open them up to other applications as well.
Web Services
SLIDER on YARN Deployment
LLAP can be deployed via Slider, which bypasses node installation and related complexities . (HIVE-9883).
Resources
Hive Contributor Meetup Presentation