You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 6 Next »

Hadoop uses some dependences that do not work natively with Apple Silicon (M1) machines. If you would like to develop and build Hadoop on your macOS with M1 chip / ARM64 CPU, follow steps in this page.

proto buffer

Hadoop 3 has moved to newer version of protobuf package (version 3.7.0 in trunk as of Mar 2021), though you can use old versions especially if you need to build both on Hadoop 2 and Hadoop 3 with one protobuf version. However, Apple M1 does not work with protobuf 2.5.0 - when installing protobuf from source you will get unsupported CPU architecture error. The simply solution is to use the same new protobuf version.

protobuf-maven-plugin

You may get following error:

[ERROR] Failed to execute goal org.xolstice.maven.plugins:protobuf-maven-plugin:0.5.1:compile-custom (default) on project hadoop-yarn-csi: Missing:
[ERROR] ----------
[ERROR] 1) io.grpc:protoc-gen-grpc-java:exe:linux-aarch_64:1.15.1

This is because the grpc package is not releasing binaries for ARM64. One way of solving this is to specify the properties os.detected.classifier to be osx-x86_64 instead of osx-aarch_64 For example, you can put following code in your ~/.m2/settings.xml file:

<settings>
    <activeProfiles>
        <activeProfile>
            apple-silicon
        </activeProfile>
    </activeProfiles>
    <profiles>
        <profile>
            <id>apple-silicon</id>
            <properties>
                <os.detected.classifier>osx-x86_64</os.detected.classifier>
            </properties>
        </profile>
    </profiles>
</settings>

frontend-maven-plugin

This plugin is used to install node.js and yarn locally. YARN-6278 added yarn-ui profile to pom.xml leveraging this plugin to automatically install node.js and yarn locally under target/webapp directory. Latest versions of this plugin supports Apple M1 chip but Hadoop is still using an older version that does not. As a result, when detecting an ARM CPU on your Apple M1, this plugin will generate a download link for a Darwin ARM64 build of Node, which doesn’t exist. So the workaround is to manually upgrade this version to 1.10+. For this you can update the version in hadoop-project/pom.xml file.

Later Hadoop release will have this plugin upgrade. Will file a JIRA for this (TBD).


  • No labels