Goal

The goal of this release is to keep enhancing Apache HAWQ features, fixing bugs. Vectorized Execution and Runtime filter will be delivered, will extend Pluggable Storage Framework to access Hive table data. As the output, both source and binary tar balls will be provided for downloading.

Plan Scope

• New Feature: Pluggable Vectorized Execution Engine on HAWQ.
• New Feature: Support Runtime Filter for HAWQ local hash join.
• New Feature: Support accessing Hive table data by Pluggable Storage Framework.
• Bug fixes. 

Release Note

Release Type

Source release and binary release.

Binary tar balls: HAWQ core, PXF.

Source tar ball: All-in-one tar ball including HAWQ core, PXF, and Ranger.

Timeline

• Target to end of June 2018 : branch cut
• Target to end of July 2018 : release

Release Version

Apache HAWQ 2.4.0.0-incubating

Apache HAWQ JIRA Release DashBoard

Sub-task

• [HAWQ-1576] - Add demo for pluggable format scan
• [HAWQ-1577] - Add demo for pluggable format insert
• [HAWQ-1583] - Add vectorized executor extension and GUC
• [HAWQ-1591] - Common tuple batch structure for vectorized execution
• [HAWQ-1592] - vectorized data types initialization and relevant function definetion
• [HAWQ-1593] - Vectorized execution condition check in plan tree
• [HAWQ-1598] - Vectorized Scan Node Framework initialization
• [HAWQ-1600] - Parquet table data vectorized scan
• [HAWQ-1601] - Vectorized Scan qualification supported
• [HAWQ-1602] - AO table data vectorized scan
• [HAWQ-1603] - add new hook api for expressions
• [HAWQ-1604] - Add A New GUC hawq_hashjoin_bloomfilter
• [HAWQ-1606] - Implement Deciding to Create Bloom Filter During Query Plan And Create Bloom filter For Inner Table
• [HAWQ-1608] - Implement Printing Runtime Filter Information For "explain analyze"

Bug

• [HAWQ-1416] - hawq_toolkit administrative schema missing in HAWQ installation
• [HAWQ-1483] - cache lookup failure
• [HAWQ-1594] - Memory leak in standby master (gpsyncagent process)
• [HAWQ-1615] - Access Invalid Memory When Run a Hash-join query with Bloomfilter Enable.
• [HAWQ-1616] - Wrong Result of Hash Join When Enable Bloom filter
• [HAWQ-1618] - Segment panic at workfile_mgr_close_file() when transaction ROLLBACK
• [HAWQ-1620] - Push Down Target List Information To Parquet Scan For Bloomfilter
• [HAWQ-1636] - Compile apache hawq failure due to unsupported syntax in libyarn on osx 10.11
• [HAWQ-1637] - Compile apache hawq failure due to Failed to execute goal org.apache.maven.plugins:maven-javadoc-plugin:2.9.1:aggregate-jar on osx 10.11

New Feature

• [HAWQ-1450] - New HAWQ executor with vectorization & possible code generation
• [HAWQ-1597] - Implement Runtime Filter for Hash Join
• [HAWQ-1599] - PXF Ignite plugin

Improvement

• [HAWQ-1621] - Call FileSystem.closeAllForUGI at the end of request when PXF impersonation is on
• [HAWQ-1622] - Cache PXF proxy UGI so that cleanup of FileSystem cache doesn't have to be done on each request
• [HAWQ-1627] - Support setting the max protocol message size when talking with HDFS
• [HAWQ-1633] - Add parameter for maven package hawq-hadoop

Wish

• [HAWQ-786] - Framework to support pluggable formats and file systems

What's fixed in the release

Process

• Collect feedback and requirements from community on the scope of release.
• Create Apache HAWQ-2.4.0.0-incubating release management page
• Create JIRA filter "HAWQ-2.4.0.0-incubating" and "HAWQ-2.4.0.0-incubating-fixed"
• Clean up all issues covered in HAWQ-2.4.0.0-incubating with fixing version 2.4.0.0
• Create Apache HAWQ-2.4.0.0-incubating release dash board
• Start release process
• Cut branch
• Send RC1 to DEV for voting
• Update websites for new release
• Announce release