You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 4 Next »

Hadoop uses some dependences that do not work natively with Apple Silicon (M1) machines. If you would like to develop and build Hadoop on your macOS with M1 chip / ARM64 CPU, follow steps in this page.

protobuf-maven-plugin

You may get following error:

[ERROR] Failed to execute goal org.xolstice.maven.plugins:protobuf-maven-plugin:0.5.1:compile-custom (default) on project hadoop-yarn-csi: Missing:
[ERROR] ----------
[ERROR] 1) io.grpc:protoc-gen-grpc-java:exe:linux-aarch_64:1.15.1

This is because the grpc package is not releasing binaries for ARM64. One way of solving this is to specify the properties os.detected.classifier to be osx-x86_64 instead of osx-aarch_64 For example, you can put following code in your ~/.m2/settings.xml file:

<settings>
<activeProfiles>
<activeProfile>
apple-silicon
</activeProfile>
</activeProfiles>
<profiles>
<profile>
<id>apple-silicon</id>
<properties>
<os.detected.classifier>osx-x86_64</os.detected.classifier>
</properties>
</profile>
</profiles>
</settings>

frontend-maven-plugin

This plugin is used to install node.js and yarn locally. YARN-6278 added yarn-ui profile to pom.xml leveraging this plugin to automatically install node.js and yarn locally under target/webapp directory. Latest versions of this plugin supports Apple M1 chip but Hadoop is still using an older version that does not. As a result, when detecting an ARM CPU on your Apple M1, this plugin will generate a download link for a Darwin ARM64 build of Node, which doesn’t exist. So the workaround is to manually upgrade this version to 1.10+.

Later Hadoop release will have this plugin upgrade. Will file a JIRA for this (TBD).


  • No labels