Fully integrate your Ranorex Report with CI tools like Jenkins using xUnit

A Ranorex test automation project creates an executable file which returns a two-state test result indicating a success or a failure. Continuous Integration (CI) tools like Jenkins, Hudson, TeamCity, CruiseControl, Bamboo etc. are able to call automated tests and do understand these two-state test results. This is similar with ALM tools like Polarion or Jira. In short, in the world of CI and ALM, test automation results are reflected by a single flag, success or failure.

This blog describes how a Ranorex Report can be automatically transformed into xUnit format by using an XSL Transformation (XSLT). This transformation allows the CI tool to fully understand the result of each test case, not only whether the entire test run passed or failed.

The following example shows how UI test results can be interpreted following the same standards as xUnit tests do. Furthermore an example implementation of an XSL transformation file can be downloaded at the end of this blog. In case there is not CI-Plugin available assisting you with the transformation you can simply implement a standalone transformation with Ranorex.

• Background
• Install xUnit Plugin for Jenkins
• Activate Custom Transformation (XSLT)
• Exploring Test Results
• Standalone Transformation (if your CI Tool does not support xUnit Transformations)
• Conclusion
• Further Reading

Background

The blog post “Integrating Ranorex Automation in Jenkins CI Process” is introducing the entire CI topic and gives a step by step introduction on how to integrate Ranorex into Jenkins CI. On the top of that, optimizations can be done supporting the machined understanding of the Ranorex Report.

Nowadays most of the modern unit testing frameworks such as JUnit, NUnit etc. do implement the xUnit notation (list of implementations). This comes with the advantage that tests and their results are defined once following the xUnit standard, and can then be interpreted and understood independent from the respective target language (such as Java, C#, etc.). In addition, continuous integration servers like Jenkins do use xUnit to interpret the results of a test in a unified way. Although Ranorex is a UI testing and not a unit testing framework, there still is a way to also interpret the Ranorex test results (Ranorex Reports) in the same way as you might be used to from your (automated) unit tests.

Install xUnit Plugin for Jenkins

First, the xUnit-plugin needs to be installed on Jenkins. Therefore please navigate to “Manage Jenkins” in the main menu and click the “Manage Plugins” menu item.

 

Please proceed by installing the plugin.

Activate Custom Transformation (XSLT)

Next, the transformation of the Ranorex Report to a xUnit-based report format has to be activated. Therefore navigate to your Ranorex project and bring up the project configuration.

Start by adding a “post build action” called “Publish xUnit test result report”.

Two new panels appear allowing to define the thresholds for failed and skipped tests (both, total and new). More importantly click the “Custom Tool” menu item in the drop down:

The file name of the Ranorex Report needs to be defined in the setting “Custom Tool Pattern”:

Note: The xUnit plugin awaits a relative path expression in the “Custom Tool Pattern”. The starting path is the Workspace.

Please make sure that path and file name of your report file pattern match the definition of your build step that calls the Ranorex executable. Here is the one used by this example:

TestCIProject.exe /rf:Report-Build-%BUILD_NUMBER%

Note: The Ranorex command line argument “rf” (reportfile) allows defining the report file name including the extension. If no file-extension is provided, the default extension “rxlog“ will be used for the report frame file. Additionally, the XML file containing the relevant data will be created with a file-extension “rxlog.data” then. For details on all command line arguments please refer to the user guide, chapter Running Tests via Command Line.

Note also: The above used placeholders %BUILD_NUMBER% and ${BUILD_NUMBER} are provided by Jenkins. At the beginning of a test run they will be replaced automatically by the current build number. Please take care to use these notations correctly.

Another relevant setting is the path to the transformation file (XSLT). A sample file is shared on GitHub and can be downloaded here:

• Based on test cases: ranorex-to-xunit.xsl (Right-click: Save-as; Recommended) or
• Based on modules: ranorex-to-xunit.xsl (Right-click: Save-as)

The entire github-repository can be found here: https://github.com/ranorex/Ranorex-To-XUnit

Given these settings, right after every Ranorex test execution triggered by Jenkins the Ranorex Report is automatically transformed into a xUnit-based XML file. This allows a deeper integration of test results into CI which will be described in the next section.

Exploring Test Results

Having applied the mentioned settings to your Ranorex project on Jenkins, a section “Test Result” will appear for every new test run.

Following the “Test Result” link, an overview site is presented providing useful information like “All Failed Tests” including their duration and age (since how many builds does test fail). Additionally, a short summary of all passed, skipped and failed tests is presented.

By clicking the plus icon of a failed test Jenkins presents error details and the stack trace additionally:

Also, Jenkins can now distinguish on the test case level whether a test case passed or failed. Before the conversion to xUnit format, Jenkins was only able to interpret the entire test result.

Standalone Transformation (if your CI Tool does not support xUnit Transformations)

In case your CI tool does not support XSL transformations and there are no plugins around assisting you, you can simply call the transformation at the end of your test run automatically.

Please follow the step by step instructions below to transform your Ranorex Report into xUnit format directly out of your Ranorex Test Suite. The resulting file can be used by any (CI) tool understanding this widely spread format.

1. Download the XSL-file (Right-Click: Save as) and save it to your Ranorex project folder. After doing so, add the file to your project (see screenshot below). 
2. Select the file (Ranorex-to-xunit.xsl) in the Projects View and hit the F4 key to open the Properties Pad. For the property “Copy to output directory”, choose “PreserveNewest” in the dropdown.
   
3. Download the Code Module “TransformReport” (Right-Click: Save as) and add it to your project (“Add existing item”, see step 1 above). This module is open source and is hosted on GitHub.
4. Assign the Module to your Test Suite and place it at the very end. Ideally you embed it in the global teardown container (additional information on teardown section).
   

Conclusion

Calling your Ranorex test automation project from a Continuous Integration (CI) environment provides a lot of benefits and allows you to represent the results of the entire build process over time. However, even more information of every single test run can be gathered out when the Ranorex Report is fully understood by the CI tool. This understanding can be achieved by transforming the test report into xUnit format. As visible above, there are only a few steps needed to see the results of every single test case instead of only one result of the entire test run. In addition these test case detail results can be used easier for further processing.

Although the sample above is using Jenkins as a CI tool this approach is the same for all CI tools understanding xUnit. Even if your CI tool does not support the transformation, the Standalone Transformation described above enables you to still make use of this powerful feature.

Further Reading

• Blog post on how to integrate Ranorex into any CI tool: http://www.ranorex.com/blog/integrate-ranorex-in-any-continuous-integration-process
• Blog post on how to integrate Ranorex Automation in Jenkins CI: http://www.ranorex.com/blog/integrating-ranorex-automation-in-jenkins-continuous-integration-process
• Github Repository with Ranorex Report xUnit Transformation File: https://github.com/ranorex/Ranorex-To-XUnit
• A nice blog post describing how a xUnit output is parsed by Jenkins: http://nelsonwells.net/2012/09/how-jenkins-ci-parses-and-displays-junit-output/
• Information about the xUnit Plugin for Jenkins: https://wiki.jenkins-ci.org/display/JENKINS/xUnit+Plugin