THIS IS A TEST INSTANCE. ALL YOUR CHANGES WILL BE LOST!!!!
Git branch: https://github.com/apache/ozone/tree/HDDS-4454
Currently, ozone write pattern is bursty and involves multiple buffers copies as well multiple ratis log syncs in a block write. The idea of the Jira is to use a zero buffer copy based Ratis streaming (RATIS-979) in ozone write path for better performance and resource utilization.
To enable the feature, the following configuration properties need to be added to Ozone Manager's ozone-site.xml.
<property>
<name>dfs.container.ratis.datastream.enable</name>
<value>false</value>
<tag>OZONE, CONTAINER, RATIS, DATASTREAM</tag>
<description>If enable datastream ipc of container.</description>
</property>
<property>
<name>ozone.fs.datastream.enabled</name>
<value>false</value>
<tag>OZONE, DATANODE</tag>
<description>
To enable/disable filesystem write via ratis streaming.
</description>
</property>
1. Builds/intermittent test failures
No additional flaky tests have been introduced by the feature branch.
2. Documentation
Documentation is being added by .
3. Design, attached the docs
The design docs can be found under the Attachments section in the umbrella jira:
4. Compatibility
Merge Ozone Streaming Write Pipeline feature does not change any existing APIs.
5. Docker-compose / acceptance tests
New acceptance test is being added by Jira:
6. Support of containers / Kubernetes:
No addition.
7. Coverage/code quality:
Current feature branch coverage is 85.0% (vs 82.3 % of master branch)
https://sonarcloud.io/summary/new_code?id=hadoop-ozone&branch=HDDS-4454
8. Build time
No significant build time difference has been observed.
master branch succeeded 3 days ago in 9m 9s: https://github.com/apache/ozone/runs/6528138840?check_suite_focus=true
Feature branch succeeded 7 days ago in
9. Possible incompatible changes/used feature flag:
There should not be any incompatible changes introduced with this feature.
A global enable/disable switch for the this feature is added in .
10. Third party dependencies/license changes:
There is no third party dependencies introduced by this feature.
11. Performance
The throughput of the new Streaming Pipeline is roughly 2x and 3x of the throughput of the existing Write Pipeline for the single client case and the 3-client case, respectively.
Single client case: 20220702_Single Client - Benchmarks for Ozone Streaming.pdf
3-client case: 20220702_3 Clients - Benchmarks for Ozone Streaming.pdf