Capturing from Apache Spark

To capture the execution of a program to BugJail, the only thing you need to do is start Java Virtual Machine with one additional option.

This article links to two alternative ways how to construct the BugJail Agent option string and then shows how to use the option to enable BugJail capturing from Apache Spark driver program or executor.

Step 1: Construct BugJail Agent option & copy to clipboard

Alternative A: construct BugJail Agent option string by using Capturing Assistant (easy user-interface, but supports only the two most common configuration options currently).

Alternative B: construct BugJail Agent option string manually (more effort, but allows using additional options).

Step 2 (A): Start Spark driver program with the BugJail Agent option

If starting with spark-submit

If you start Spark by using the spark-submit -script, you can enable BugJail capturing for the driver program by adding the following option to spark-submit command-line:

--driver-java-options="<bugjail_agent_option>"

If starting in-process

If you start Spark "in-process" from another program (command line or during development from an IDE), you can enable BugJail capturing just the same way as for normal condole / command-line programs, or when capturing from IDEs.

Step 2 (B): Start Spark executor with the BugJail Agent option

If starting with spark-submit

If you start Spark by using the spark-submit -script, you can enable BugJail capturing for the executors by adding the following option to spark-submit command-line:

--conf "spark.executor.extraJavaOptions=<bugjail_agent_option>"

Note: be sure that the agent.jar is available in the machines where the executors run!

TODO add instructions for distributing the agent.jar

If starting in-process

If you start Spark "in-process" with another program, the executor is actually the same process as the driver, and BugJail capturing is enabled same way as for normal condole / command-line programs, or when capturing from IDEs.