Page History

...

4. Configure the Spark JVMs to use the YourKit profiling agent by editing `~/spark/conf/spark-env.sh` and adding the lines```

SPARK_DAEMON_JAVA_OPTS+=" -agentpath:/root/yjp-12.0.5/bin/linux-x86-64/libyjpagent.so=sampling"

...

export SPARK_DAEMON_JAVA_OPTS

...

SPARK_JAVA_OPTS+=" -agentpath:/root/yjp-12.0.5/bin/linux-x86-64/libyjpagent.so=sampling"

...

export SPARK_JAVA_OPTS

...

5. Copy the updated configuration to each node: `~/spark-ec2/copy-dir ~/spark/conf/spark-env.sh`

6. Restart your Spark cluster:```

~/spark/bin/stop-all.sh

...

~/spark/bin/start-all.sh

...

7. By default, the YourKit profiler agents use ports 10001-10010. To connect the YourKit desktop application to the remote profiler agents, you'll have to open these ports in the cluster's EC2 security groups.

...