Page History

Here are instructions on profiling Spark applications using YourKit Java Profiler.

On Spark EC2 images

1. After logging into the master node, download the YourKit Java Profiler for Linux from the YourKit downloads page (at the time of writing, the latest version is yjp-12.0.5-linux.tar.bz2; you will need to substitute different paths if using a newer version). This file is pretty big (~100 MB) and YourKit downloads site is somewhat slow, so you may consider mirroring this file or including it on a custom AMI.

2. Untar this file somewhere (in `/root` in our case):

 tar xvjf yjp-12.0.5-linux.tar.bz2

3. Copy the expanded YourKit files to each node using copy-dir:

~/spark-ec2/copy-dir /root/yjp-12.0.5

4. Configure the Spark JVMs to use the YourKit profiling agent by editing `~/spark/conf/spark-env.sh` and adding the lines

SPARK_DAEMON_JAVA_OPTS+=" -agentpath:/root/yjp-12.0.5/bin/linux-x86-64/libyjpagent.so=sampling"
export SPARK_DAEMON_JAVA_OPTS
SPARK_JAVA_OPTS+=" -agentpath:/root/yjp-12.0.5/bin/linux-x86-64/libyjpagent.so=sampling"
export SPARK_JAVA_OPTS

5. Copy the updated configuration to each node:

~/spark-ec2/copy-dir ~/spark/conf/spark-env.sh

6. Restart your Spark cluster:

~/spark/bin/stop-all.sh
~/spark/bin/start-all.sh

...

Please see the full YourKit documentation for the full list of profiler agent startup options.

In Spark unit tests

When running Spark tests through SBT, use

export java_opts=" -agentpath:/path/to/yjp/agent/"

to launch the tests with the YourKit profiler agent enabled.  The platform-specific paths to the profiler agents are listed in the YourKit documentation.  Note that java_opts is in lower-case.