You are viewing an old version of this page. View the current version.

Compare with Current View Page History

Version 1 Next »

Git branch: https://github.com/apache/ozone/tree/HDDS-4454


Currently, ozone write pattern is bursty and involves multiple buffers copies as well multiple ratis log syncs in a block write. The idea of the Jira is to use a zero buffer copy based Ratis streaming (RATIS-979) in ozone write path for better performance and resource utilization.

To enable the feature, the following configs need to be added to Ozone Manager's ozone-site.xml.

  <property>
    <name>dfs.container.ratis.datastream.enable</name>
    <value>false</value>
    <tag>OZONE, CONTAINER, RATIS, DATASTREAM</tag>
    <description>If enable datastream ipc of container.</description>
  </property>


1. Builds/intermittent test failures

No additional flaky tests have been introduced by the feature branch.

2. Documentation

Documentation is being added by  Unable to render Jira issues macro, execution error. . 

3. Design, attached the docs

The design docs can be found under the Attachments section in the umbrella jira:  Unable to render Jira issues macro, execution error.

4. Compatibility

Merge Ozone Streaming Write Pipeline feature does not change any existing APIs.

5. Docker-compose / acceptance tests

New acceptance test is being added by Jira: Unable to render Jira issues macro, execution error.

6. Support of containers / Kubernetes:

No addition.

7. Coverage/code quality: 

Current feature branch coverage is 85.0% (vs 82.3 % of master branch)


https://sonarcloud.io/summary/new_code?id=hadoop-ozone&branch=HDDS-4454

8. Build time

No significant build time difference has been observed.

master branch succeeded 3 days ago in 9m 9s: https://github.com/apache/ozone/runs/6528138840?check_suite_focus=true

Feature branch succeeded 7 days ago in 8m 42s: https://github.com/apache/ozone/runs/6445840932?check_suite_focus=true

9. Possible incompatible changes/used feature flag: 

There should not be any incompatible changes introduced with this feature.

A global enable/disable switch for the this feature is added in Unable to render Jira issues macro, execution error. .

10. Third party dependencies/license changes:

There is no third party dependencies introduced by this feature.

11. Performance

We have tested major datanode activities which require RocksDB operation, include container create & close & delete, and block put & get.  Except that container delete performance drops because container metadata KV need to be deleted from RocksDB, other four major activities all have performance improved. 


  • No labels