Overview

The command line code coverage tool is used to gather and report code coverage statistics during nightly builds where using an IDE tool is not possible. The tool consists of three different pieces:

1. Flex preload SWF
2. Java Code Coverage Server
3. Java Code Coverage Reporter

The Flex portion of the tool is a SWF that is preloaded for every SWF that is run. The job of the preload SWF is to listen to the execution of lines in the running SWF. Every executed line is sent over a socket to the Java server. The Java server will write the data to disk where it can be analyzed later. After all of the SWFs have been run the Java server is shutdown to flush all the data to disk.

After the execution data has been collected the Code Coverage Reporter can be run to analyze the results. The reporting tool provides a summary of the total line coverage and total method coverage. By default, more detailed information can be found at the package, file, and method level. At the package level the following information is available:

1. Total lines executed in the package, total lines in the package, and percent coverage.
2. Total methods executed in the package, total methods in the package, and percent coverage.
3. List of files in the package (see file level details below).

At the file level the following information is available:

1. Total lines executed in the file, total lines in the file, and percent coverage.
2. Total methods executed in the file, total methods in the file, and percent coverage.
3. List of unexecuted lines and unexecuted methods.
4. List of methods in the file.

At the method level the following information is available:

1. Total lines executed in the method, total lines in the method, and percent coverage.

How to Use

All of the Code Coverage Tool's command files are located in its "bin" directory. It is recommended to add this directory to your path. The following commands are available on either the Windows or Unix platforms:

For more information on the command line options and configuration options see the Code Coverage Server and Code Coverage Reporter sections below.

The Code Coverage Server is started before running of any tests. After the tests have completed the Code Coverage Server is shut down to flush all data to disk. The tests can be either flexunit or mustella tests or a combination of both. After all the data has been collected one or more reports can be created.

Example

This example will show how to run code coverage on a mustella test. It assumes the Code Coverage Tool's "bin" directory in on the path and the user is located in the mustella directory. First the Java server is started in the background:

$ start-ccserver

$ Apache Flex Code Coverage Server

Version 0.9

listening on port 9097

Next the mustella test is run:

>./mini_run.sh components/Button

After the mustella test finishes the Code Coverage Server is shutdown:

$ ccserver stop

Apache Flex Code Coverage Server

Version 0.9

stopping Apache Flex Code Coverage Server

Now that the raw coverage data has been captured we can generate a report to see the coverage of mx.controls.Button. The overall coverage summary is output to the console and the details are written to the report file.

$ ccreporter -include mx.controls.Button

Apache Flex Code Coverage Reporter

Version 0.9

Found 3 files in /Users/dloverin/ccdata

Analyzing 3 files

Results:

Line Coverage:   78%

Method Coverage: 77%

Report output to /Users/dloverin/ApacheFlex/sdk/mustella/ccreport.xml

Now ccreport.xml can be examined to look at the unexecuted lines and see how the tests can be improved.

If code coverage is run on multiple packages using the "-hideFiles" option is recommended to reduce the size of the report. Once packages with low coverage are found the report can be regenerated with full details but filtered to only include the classes of concern.

Note on Mustella Timeouts

When mustella tests are running under code coverage, cpu intensive tests will run slower than normal. If any of your mustella tests explicitly set the timeout attribute it is recommended to use the mustella timeout options to effectively double that timeout. For example, if you have a test that has a timeout of 5 seconds, set the mustella -set_timeout option to 5 seconds which will add 5 seconds to the timeout of every test. Setting -timeout can also be useful to prevent timeout failures:

 >./mini_run.sh -timeout=10 -set_timeout=5 -all

How it works

When the Code Coverage Server (ccserver) is started it modifies the user's mm.cfg file, adding the preloadSWF entry. The preloadSWF entry is set to the path of the CodeCoveragePreloadSWF.swf in the Code Coverage's Tool lib directory. The ccserver then waits for the preload SWF to connect with it over the data port. Once a connection is established the server creates a thread to read the trace data from the preload SWF and writes the data to a file. When the next preload SWF makes a connection the previous thread is stopped and the data file is closed before create a new thread and a new data file. Each SWF that is run results in a new data file in the data directory.

The preload SWF initiates a stream of trace data from the Flash Player when it is loaded. The trace data includes the paths, package, file, and line number that is executed. The trace information is optimized and written over a socket to the ccserver. Two optimizations are performed by the Flex client:

1. Id numbers are generated for the debug file strings. The id and debug file string are sent once to the server and afterwards only the id is sent.
2. Executed lines are only sent once to the server. For the purposes of this tool we only care if a line is execute or not, not how many times it is executed.

The ccserver listens for commands on the command port. When it receives the "stop" command it closes any existing data files and exits.

 After the ccserver has shutdown the Code Coverage Reporter can be run to read the data files and create a report. The Code Coverage Reporter reads a data file to get the executed lines and also reads the executed SWFs to get the total number of lines that could be executed. This allows the reporter to compare the executed lines vs. the total number of lines. If any of the SWFs have been deleted before the Reporter is run, the report will failure since that total number of lines will cannot be calculated.

Code Coverage Data File Format

Code Coverage Server

The Code Coverage Server's primary job is to write code coverage data to the data directory. The server performs four different functions:

1. Write code coverage data to the data directory.
2. At start up time this server is responsible for modifying the user's mm.cfg file so a preload SWF will be loaded for every SWF that is run.
3. Provide a policy file to the Flash Player to allow code coverage data to be sent from the preload SWF to the server over the data port.
4. Accept commands on the command port. The server is shutdown by sending it a stop command over the command port.

The format of the command line is:

ccserver [start | stop]

start_ccserver

The server can be started with either the "start_ccserver" command or the "ccserver start" command. The difference is the "start_ccserver" starts the server in the background so the command line is not blocked. 

The server is shutdown with the "ccserver stop" command.

Command Line Options

Configuration options

The ccserver.properties file is located in the Code Coverage Tool's lib directory.

Code Coverage Reporter

The Code Coverage Reporter can be run without command line options if the ccserver is also run with the default configuration.

The format of the command line is:

ccreporter [options]* [filename | directory]*

The command is followed by zero or options and ends with a list of files and directory to read code coverage data files from. If the list of files and directory is not provided then the reports looks in the "ccdata" directory in the user's home directory (also default location of the ccserver).

Command line options