...
In this version, we add support for HDFS as well as the rate-limiting of HDFS downloading/uploading.
HTTP APIs of the Bulk-Load procedure
Bulk Load in version 2.1.0 is relatively primitive. We make this feature full-feathered in this version with a list of APIs that can help us to automate the procedure.
(We in Xiaomi have developed a tool called BulkLoad Manager that simplifies the management of tasks. It utilizes the newly exposed APIs. We plan to open-source this project soon.)
Fixed Issues
From this version, we have support for various C++ compilers, including:
- GCC 5.4.0 (ubuntu1604)
- GCC 7.5.0 (ubuntu1804)
- GCC 9.4.0 (ubuntu2004)
- Clang9
- Clang10
The continuous testing is here: https://github.com/pegasus-kv/pegasus-docker
Known Issues
...
Apache Pegasus 2.2.0 是一个功能版本。所有的改动都被总结在: https://github.com/apache/incubator-pegasus/issues/696。
...
支持HDFS用于BulkLoad/Backup/Restore
在我们绝大多数的BulkLoad用户场景,HDFS都是Spark生成文件的默认存储。在这一版本中,我们提供了HDFS的支持,以及HDFS文件上传下载限流的支持。
Bulk Load流程的HTTP接口
2.1.0版本的Bulk Load是相对初级的。我们在这一版本中将该功能打造得更为完整。我们提供了一系列API,它们可以用于BulkLoad流程的自动化。
(在小米,我们开发了BulkLoad Manager运维工具,它简化了BulkLoad的任务管理。它使用了该版本引入的新API。我们计划近期将该工具开源。)在我们绝大多数的BulkLoad用户场景,HDFS都是Spark生成文件的默认存储。在这一版本中,我们提供了HDFS以及HDFS文件上传下载的支持。