Page History

...

To enable the feature, the following configs configuration properties need to be added to Ozone Manager's ozone-site.xml.

  <property>
    <name>dfs.container.ratis.datastream.enable</name>
    <value>false<<value>true</value>
    <tag>OZONE, CONTAINER, RATIS, DATASTREAM</tag>
    <description>If enable datastream ipc of container.</description>
  </property>

  <property>
    <name>ozone.fs.datastream.enabled</name>
    <value>true</value>
    <tag>OZONE, DATANODE</tag>
    <description>
      To enable/disable filesystem write via ratis streaming.
    </description>
  </property> 

1. Builds/intermittent test failures

...

2. Documentation

Documentation is being was added by 

3. Design, attached the docs

The design docs can be found under the Attachments section in the umbrella jira: 

4. Compatibility

Merge Ozone Streaming Write Pipeline feature does not change any existing APIs.

5. Docker-compose / acceptance tests

New acceptance test is being was added by Jira:

6. Support of containers / Kubernetes:

...

7. Coverage/code quality: 

Current HDDS-4454 feature branch coverage is 85.0% (vs 82.3 % of master branch)https://sonarcloud.io/summary/new_code?id=hadoop-ozone&branch=HDDS-4454 (65.4%) vs master branch coverage (65.9%)

8. Build time

No significant build time difference has been observed.

master branch succeeded 3 days ago in 9m 9s(1h 28m 9s): https://github.com/apache/ozone/actions/runs/6528138840?check_suite_focus=true3428272225

HDDS-4454 branch succeeded (1h 22m 11s)Feature branch succeeded 7 days ago in 8m 42s: https://github.com/apache/ozone/actions/runs/6445840932?check_suite_focus=true3412888917

9. Possible incompatible changes/used feature flag: 

...

A global enable/disable switch for the this feature is added in

10. Third party dependencies/license changes:

There is no third party dependencies introduced by this feature.

11. Performance

We have tested major datanode activities which require RocksDB operation, include container create & close & delete, and block put & get.  Except that container delete performance drops because container metadata KV need to be deleted from RocksDB, other four major activities all have performance improved. 

The throughput of the new Streaming Pipeline is roughly 2x and 3x of the throughput of the existing Write Pipeline for the single client case and the 3-client case, respectively.

Single client case: 20220702_Single Client - Benchmarks for Ozone Streaming.pdf

3-client case: 20220702_3 Clients - Benchmarks for Ozone Streaming.pdfImage Removed