Jump to: navigation, search

CrossProject:HouseKeeping Best Practices Group:Integration Test

Introduction

Prior to Karaf, pax-exam was proven to be a very fragile integration test mechanism, often failing because of changes that only impact pax-exam, and not the actual system being tested.


Other Proposed Solutions

  1. CONTROLLER-LEVEL-INTEGRATION: There is a controller-maven-plugin coming in that should allow the running of integration tests in a real running controller, rather than an artificial pax-exam environment:

Gerrit 3420. Thus allowing for both more realistic and also more robust integration testing.

Running UT and Getting JaCoCo Code Coverage with Pax Exam and Junit

The UT JaCoCo settings are enabled in odlparent and the following sections are no longer required in local parent poms. All ODL projects now have UT coverage reporting enabled by default via odlparent.

Jump to the IT configuration below to configure IT related test for code coverage which needs to be configured at the integration test artifact level.

Running IT and Getting JaCoCo Code Coverage via Pax Exam and Junit

project structure

To avoid circular dependencies, integration tests should be set up as a separate module that runs after the module that builds karaf features for a project.

configuring the IT project

When running IT test cases in a separate project, things get a bit more tricky. First, we need to specify where the itReport will be found, which is done by setting a parameter (Note, this all assumes that integration test inherits from feature parent, not parent...)

 <properties>
   <sonar.jacoco.itReportPath>target/jacoco-it.exec</sonar.jacoco.itReportPath>
 </properties>

Next, we have to get plugins in the correct order. The first plugin to specify is maven-paxexam-plugin, which generates a dependency file. This allows the test code to use projectAsInVersion(), which ensures portability:

     <plugin>
       <groupId>org.ops4j.pax.exam</groupId>
       <artifactId>maven-paxexam-plugin</artifactId>
       <executions>
         <execution>
           <id>generate-config</id>
           <goals>
             <goal>generate-depends-file</goal>
           </goals>
         </execution>
       </executions>
     </plugin>

Next is the maven-failsafe-plugin. While the surefire plugin handles unit tests, the failsafe plugin runs all integration tests and only afterwards checks for test validity:

     <plugin>
       <artifactId>maven-failsafe-plugin</artifactId>
       <version>${failsafe.version}</version>
       <executions>
         <execution>
           <id>integration-tests</id>
           <phase>integration-test</phase>
           <goals>
             <goal>integration-test</goal>
             <goal>verify</goal>
           </goals>
           <configuration>
             <argLine>${failsafeArgLine}</argLine>
             <skipTests>${skip.integration.tests}</skipTests>
           </configuration>
         </execution>
       </executions>
     </plugin>

Now, to make jacoco code coverage work, we need to ensure that all dependent classes and source code are in the IT project. We do this by configuring the maven dependency plugin to unpack dependent classes under target/classes:

     <plugin>
       <groupId>org.apache.maven.plugins</groupId>
       <artifactId>maven-dependency-plugin</artifactId>
       <executions>
         <execution>
           <id>unpack-classes</id>
           <goals>
             <goal>unpack</goal>
           </goals>
           <configuration>
             <artifactItems>
               <artifactItem>
                 <groupId>${project.groupId}</groupId>
                 <artifactId>neutron-spi</artifactId>
                 <version>${project.version}</version>
               </artifactItem>
               <artifactItem>
                 <groupId>${project.groupId}</groupId>
                 <artifactId>northbound-api</artifactId>
                 <version>${project.version}</version>
               </artifactItem>
               <artifactItem>
                 <groupId>${project.groupId}</groupId>
                 <artifactId>transcriber</artifactId>
                 <version>${project.version}</version>
               </artifactItem>
             </artifactItems>
             <outputDirectory>target/classes</outputDirectory>
           </configuration>
         </execution>
       </executions>
     </plugin>

Second, we configure the antrun plugin to copy necessary source code to target/generated-sources/dependency during the pre-integration-test phase and then to remove this code during the verify phase - this removal is necessary as otherwise the effective size of the project doubles in sonar:

     <plugin>
       <artifactId>maven-antrun-plugin</artifactId>
       <executions>
         <execution>
           <id>prep-jacoco-agent</id>
           <phase>pre-integration-test</phase>
           <goals>
             <goal>run</goal>
           </goals>
           <configuration>
             <target>
               <copy todir="target/generated-sources/dependency" overwrite="true">
                 <fileset dir="../../neutron-spi/src/main/java" casesensitive="yes" />
                 <fileset dir="../../northbound-api/src/main/java" casesensitive="yes" />
                 <fileset dir="../../transcriber/src/main/java" casesensitive="yes" />
               </copy>
             </target>
           </configuration>
         </execution>
         <execution>
           <id>remove-generated-sources</id>
           <phase>verify</phase>
           <goals>
             <goal>run</goal>
           </goals>
           <configuration>
             <target>
               <delete includeEmptyDirs="true">
                 <fileset dir="target/generated-sources/dependency" includes="**/*" defaultexcludes="no"/>
               </delete>
             </target>
           </configuration>
         </execution>
       </executions>
     </plugin>

Lastly, we configure the jacoco plugin itself:

     <plugin>
       <groupId>org.jacoco</groupId>
       <artifactId>jacoco-maven-plugin</artifactId>
       <executions>
         <execution>
           <id>pre-integration-test</id>
           <goals>
             <goal>prepare-agent-integration</goal>
           </goals>
           <configuration>
             <destFile>${sonar.jacoco.itReportPath}</destFile>
           </configuration>
         </execution>
         <execution>
           <id>post-integration-test</id>
           <phase>post-integration-test</phase>
           <goals>
             <goal>report-integration</goal>
           </goals>
           <configuration>
             <dataFile>${sonar.jacoco.itReportPath}</dataFile>
             <outputDirectory>${project.basedir}/target/site/jacoco-it</outputDirectory>
           </configuration>
         </execution>
       </executions>
     </plugin>

configuring the parent pom

The following property needs to be set in the parent pom as it tells sonar where to find the IT report from jacoco for all other projects:

   <sonar.jacoco.itReportPath>../target/jacoco-it.exec</sonar.jacoco.itReportPath>

configuring the top level pom

For sonar reporting of IT to work, the top level pom needs to ensure that the jacoco reports are available from the directory given via the parent pom.

We do this be configuring the antrun task to move things during the verify phase:

     <plugin>
       <artifactId>maven-antrun-plugin</artifactId>
       <executions>
         <execution>
           <id>move-it-reports</id>
           <phase>verify</phase>
           <goals>
             <goal>run</goal>
           </goals>
           <configuration>
             <target>
               <move todir="target/site">
                 <fileset dir="integration/test/target/site" />
               </move>
               <move file="integration/test/target/jacoco-it.exec" todir="target" />
             </target>
           </configuration>
         </execution>
       </executions>
     </plugin>

Running IT and Getting JaCoCo Code Coverage via Karaf containers within Pax Exam and Junit

Now that OpenDaylight uses karaf containers, it is much easier to set up a "real running controller" for integration tests. This section talks about how to modify the above configuration to do this...

IT pom.xml modifications

Because we are going to attach jacoco to the forked VM started by pax-exam that will run the system under test, it is necessary to specify a dependency against the jacoco agent:

   <dependency>
     <groupId>org.jacoco</groupId>
     <artifactId>org.jacoco.agent</artifactId>
     <version>${jacoco.version}</version>
   </dependency>

While the dependency to jacoco above ensures that the artifact is downloaded into the local maven repository, we want then runtime jar that is part of that artifact to be available to pax-exam for attaching to the forked VM. So, we add another target to the antrun plugin to copy the runtime jar to a path within the IT project as part of the pre-integration-test step of the build lifecycle. Further, we will want the jacoco coverage file generated within the forked VM to be available for reporting to Sonar, we add a post-integration-test phase target that copies out the jacoco file. The reason for doing this is that pax-exam adds a random UUID to the path used to root the forked VM and it is easier to configure the jacoco plugin to report from a fixed location. Lastly the root of the paths (target/pax) is what will be specified to pax-exam as the location to unpack and run the karaf distribution. The modified plugin configuration looks like:

     <plugin>
       <artifactId>maven-antrun-plugin</artifactId>
       <executions>
         <execution>
           <id>prep-jacoco-agent</id>
           <phase>pre-integration-test</phase>
           <goals>
             <goal>run</goal>
           </goals>
           <configuration>
             <target>
               <copy file="${settings.localRepository}/org/jacoco/org.jacoco.agent/${jacoco.version}/org.jacoco.agent-${jacoco.version}-runtime.jar"
                     tofile="target/pax/jars/org.jacoco.agent.jar" />
               <copy todir="target/generated-sources/dependency" overwrite="true">
                 <fileset dir="../../neutron-spi/src/main/java" casesensitive="yes" />
                 <fileset dir="../../northbound-api/src/main/java" casesensitive="yes" />
                 <fileset dir="../../transcriber/src/main/java" casesensitive="yes" />
               </copy>
             </target>
           </configuration>
         </execution>
         <execution>
           <id>copyout-coverage-file</id>
           <phase>post-integration-test</phase>
           <goals>
             <goal>run</goal>
           </goals>
           <configuration>
             <target>
               <copy todir="${project.basedir}/target" flatten="true" overwrite="true">
                 <fileset dir="target/pax" casesensitive="yes">
                   <include name="**/jacoco-it.exec" />
                 </fileset>
               </copy>
             </target>
           </configuration>
         </execution>
         <execution>
           <id>remove-generated-sources</id>
           <phase>verify</phase>
           <goals>
             <goal>run</goal>
           </goals>
           <configuration>
             <target>
               <delete includeEmptyDirs="true">
                 <fileset dir="target/generated-sources/dependency" includes="**/*" defaultexcludes="no"/>
               </delete>
             </target>
           </configuration>
         </execution>
       </executions>
     </plugin>

In the case of the jacoco plugin, it is no longer necessary to prepare the agent, as the coverage file will be written from within the forked VM. Therefore, we can remove the execution that runs during the pre-integration-test phase:

     <plugin>
       <groupId>org.jacoco</groupId>
       <artifactId>jacoco-maven-plugin</artifactId>
       <configuration>
         <includes>
           <include>org.opendaylight.neutron.*</include>
         </includes>
       </configuration>
       <executions>
         <execution>
           <id>post-integration-test</id>
           <phase>post-integration-test</phase>
           <goals>
             <goal>report</goal>
           </goals>
           <configuration>
             <dataFile>${project.build.directory}/coverage-reports/jacoco-it.exec</dataFile>
             <outputDirectory>${project.reporting.outputDirectory}/jacoco-it</outputDirectory>
           </configuration>
         </execution>
       </executions>
     </plugin>

Integration Test Java Class

For the integration test class, the following static imports will be used for pax exam and junit:

 import static org.ops4j.pax.exam.CoreOptions.maven;
 import static org.ops4j.pax.exam.CoreOptions.vmOption;
 import static org.ops4j.pax.exam.karaf.options.KarafDistributionOption.configureConsole;
 import static org.ops4j.pax.exam.karaf.options.KarafDistributionOption.debugConfiguration;
 import static org.ops4j.pax.exam.karaf.options.KarafDistributionOption.features;
 import static org.ops4j.pax.exam.karaf.options.KarafDistributionOption.karafDistributionConfiguration;
 import static org.ops4j.pax.exam.karaf.options.KarafDistributionOption.keepRuntimeFolder;
 import static org.ops4j.pax.exam.karaf.options.KarafDistributionOption.logLevel;
 import static org.ops4j.pax.exam.karaf.options.LogLevelOption.LogLevel;
 
 import java.io.File;
 import javax.inject.Inject;
 
 import org.junit.Assert;
 import org.junit.Test;
 import org.junit.runner.RunWith;
 
 import org.ops4j.pax.exam.Option;
 import org.ops4j.pax.exam.Configuration;
 import org.ops4j.pax.exam.junit.PaxExam;
 
 import org.osgi.framework.BundleContext;
 import org.osgi.service.cm.ConfigurationAdmin;

The public class gets annotated with @RunWith to ensure pax-exam is used and bundleContext and configurationAdmin objects are annotated to be injected:

 @RunWith(PaxExam.class)
 public class ITNeutronE2E {
 
     @Inject
     private BundleContext bundleContext;
 
     @Inject
     private ConfigurationAdmin configurationAdmin;

Karaf and forked VM options are configured via a config() method that is annotated as holding @Configuration. The first option specifies the karaf container and which feature should be installed. The unpack directory is set to target/pax (to match up with how the pom.xml is configured. Next is an option that specifies that jacoco is to be attached to the forked vm and where the code coverage file should be placed. The path to the jacoco agent jar must match the destination used for the antrun task run in the pre-integration-test phase, with the change that while the antrun task path is rooted at the project build directory, the forked VM will be running in a directory that is the unpackDirectory plus a random UUID (hence the ../ prefix). After the VM option, we specify that the run time folder should be kept around, that we can ignore the console and that the logLevel should default to INFO (keeping the run time folder around is very useful when developing locally, and doesn't clutter up jenkins as the entire project build directory in jenkins is ephermeal:

     @Configuration
     public Option[] config() {
         return new Option[] {
             karafDistributionConfiguration()
                 .frameworkUrl(
                     maven()
                         .groupId("org.opendaylight.neutron")
                         .artifactId("neutron-karaf")
                         .type("zip")
                         .versionAsInProject())
                 .karafVersion("3.0.3")
                 .name("Neutron")
                 .unpackDirectory(new File("target/pax"))
                 .useDeployFolder(false),
             vmOption("-javaagent:../jars/org.jacoco.agent.jar=destfile=jacoco-it.exec"),
             keepRuntimeFolder(),
             configureConsole().ignoreLocalConsole(),
             logLevel(LogLevel.INFO)
        };
     }

At this point, you can specify your test class, annotated with @Test:

     @Test
     public void test() {
     [...]
     }
 }

Happy testing!