• Created by Unknown User (elek) on Sep 30, 2019 2 minute read

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

Version 1 Next »

As discussed on the mailing list:

When Ozone was adopted as a new Hadoop subproject it was proposed[1] to 
be part of the source tree but with separated release cadence, mainly 
because it had the hadoop-trunk/SNAPSHOT as compile time dependency.

During the last Ozone releases this dependency is removed to provide 
more stable releases. Instead of using the latest trunk/SNAPSHOT build 
from Hadoop, Ozone uses the latest stable Hadoop (3.2.0 as of now).

As we have no more strict dependency between Hadoop trunk SNAPSHOT and 
Ozone trunk I propose to separate the two code base from each other with 
creating a new Hadoop git repository (apache/hadoop-ozone.git):

With moving Ozone to a separated git repository:

  * It would be easier to contribute and understand the build (as of now 
we always need `-f pom.ozone.xml` as a Maven parameter)
  * It would be possible to adjust build process without breaking 
Hadoop/Ozone builds.
  * It would be possible to use different Readme/.asf.yaml/github 
template for the Hadoop Ozone and core Hadoop. (For example the current 
github template [2] has a link to the contribution guideline [3]. Ozone 
has an extended version [4] from this guideline with additional 
information.)
  * Testing would be more safe as it won't be possible to change core 
Hadoop and Hadoop Ozone in the same patch.
  * It would be easier to cut branches for Hadoop releases (based on the 
original consensus, Ozone should be removed from all the release 
branches after creating relase branches from trunk)

Technical solution

There are two main methods to split repository:

(1) Keep the git history and push the last commit to a separatede repository

This approach doesn't require any special attention we can push the existing source to a separated repository (only the selected branches).

Advantages:

  1. Git log/history will contain all the irrelevant commits (ancient mapreduce/yarn commits for example)
  2. Repo size will remain very huge (~1G for Hadoop) which makes slower all of the CI steps

Disadvantages:

  1. Easier to search for 5 years old changes


(2) Filter branch and create a new git history

This approach is more trikcy. In git, we can freely adjust the git history and remove any commits from the history. Practically it means that we would keep only the history of hadoop-ozone and hadoop-hdds subdirectories.

Advantages:

  1. Clean, shorter, meaningful history
  2. Smaller size (1542 HDDS Jira, 15Mb vs the 1g of Hadoop)

Disadvantages

  1. All the changes before the commit "HDFS-13258. Ozone: restructure Hdsl/Ozone code to separated maven subprojects." will be available only from the Hadoop repository.

See this repository as an example: HDFS-13258. Ozone: restructure Hdsl/Ozone code to separated maven subprojects.

Note: I suggest to follow the second approach

Branching

As this is a new repository we don't need to follow the Hadoop "trunk" historical naming convention. We can go with the master which makes it easier to use it from external tools.

  • No labels