Integrating Ranorex with Testlink and Jenkins

This blog post is heavily based on ideas and instructions provided by Timm Groeger from Buhl Data GmbH.

Testlink is a very powerful test management tool which is web based and open source. In addition to support for test cases, test suites, test plans and test projects, it also supports management for feature requirements and test specifications. Furthermore, an interface for defect/issue management (e.g. Jira, Bugzilla, Mantis, etc.) exists in the framework of TestLink. Test cases can be marked as manual or automated.

Following the instructions below allows you to incorporate test management with automated test execution controlled by a CI tool (e.g. Jenkins) and executed by Ranorex. Afterwards the test results will be provided for the test management tool. This seamless integration allows traceability from beginning to end – i.e. from setting the requirements to all assigned test cases, including their (automated) test results and reports.

• Prerequisites – Tools
• TestLink Configuration
• Jenkins Configuration
• Test Execution and Result Inspection
• Future Test Cases
• Conclusion

Prerequisites – Tools

TestLink requires a web server with PHP and database access. This environment can be set up within a few minutes, for example with a WAMP installer (Windows, Apache, MySQL, PHP). TestLink can easily be placed on the top of this framework .

The widely used continuous integration tool Jenkins provides a plug-in for TestLink. Also, other continuous integration tools might provide native or plug-in based support for this test management tool.

TestLink Configuration

As a first step, a channel for the communication between TestLink, Jenkins and Ranorex needs to be defined. This channel can be generated using “Custom Fields” in TestLink (System > Define Custom Fields).

This custom field needs to be assigned to a test project in order to act as a channel for communication (Test Project > Assign Custom Fields).

As a preparation to linking the test cases from TestLink to test case results from the (automated) test runs, the type has to be set to “Automated” in the test case. Additionally, the value of the custom field has to be set to the test case name.

Note: This value has to be exactly the same (case-sensitive) as the name of the test case in Ranorex! This exact match is necessary for the TestLink plug-in to read out the test case result from the Ranorex report and to transfer it to the corresponding TestLink test case.

In the test case to be automated, the type has to be set to “Automated”. Additionally, the custom field “CommunicationChannel (comChannel)” is set to “SmokeTest”.

In the Ranorex test suite, the test case “SmokeTest” could look like the picture below.

Jenkins Configuration

A Jenkins project usually contains a couple of build steps. Typically these include:

• Building the application under test
• Building the test automation project (Ranorex solution)
• Running the test automation

(See the following Ranorex blog post for more detailed information: Integrating Ranorex Automation in Jenkins CI Process)

Invoke TestLink

The common build steps mentioned above need to be extended by adding a new build step “Invoke TestLink”.

Please provide all required information from your TestLink project here.

Note: The “Custom Fields” information needs to match the name (not label) of the custom field in TestLink. In this example, the custom field is called “CommunicationChannel”.

In the subsection “Test Execution” of the “Invoke TestLink” area, the usual steps to call test automation need to be provided.

Note: TestLink in the screenshot above is the name of the Ranorex solution and TestLink.exe is the Ranorex executable. The first line changes the directory to the output folder where the executable is located, whereas the second line simply calls the executable and defines the name of the report file (more information on command line parameters).

Transforming the Ranorex Report to an xUnit Report

After test execution, the generated Ranorex report (RanorexReport.rxlog) needs to be transformed into a format that can be understood by the TestLink plug-in (e.g. xUnit). This transformation can be done easily using the command line tool from Microsoft (msxsl.exe) and a transformation stylesheet (e.g. ranorex-to-xunit.xsl) in the following way:

Msxsl.exe <ReportFile> <TransformationStyleSheet> -o <OutputFile>

Simply add another “single build step” (execute Windows batch command) and perform the transformation like this:

Writing Results Back to TestLink

The final step is to provide TestLink with the automated test results; this can be implemented using a “Result Seeking Strategy”.

In the field “Include Pattern” the file name of the xUnit-based report file needs to be provided. The “Key Custom Field” needs to be set in the field above. The value of this field is used to map the test case result (from the xUnit report file) to the equivalent test case in TestLink. Please be sure to use the same name as defined in TestLink.

Test Execution and Result Inspection

In Jenkins, builds can be triggered in various ways: manually, periodically or polled via source code management. In the example below, a specific build is triggered manually by using the “Build Now” context menu:

No matter how the build is initiated, after the build step to run the test automation, the Ranorex report will be available.

The following build step (see Transform Ranorex Report to an xUnit Report) transforms the Ranorex report into an xUnit-based report. This transformation lets the TestLink plug-in understand and process the desired result-information.

In a final step, the test results are written back  and stored in TestLink. The screenshot below shows the execution history of the test case “SmokeTest” from the sample above, including a link to the xUnit based report file.

In the section “report” in TestLink, enhanced metrics and statistics can be generated and test result information over the time is accessible.

Future Test Cases

For test cases generated in the future, the following condition needs to be fulfilled:

In the test case representation in TestLink, the custom field needs to be set with the name of the Ranorex Test Case.

Conclusion

Tools for test management, continuous integration and test automation can be very powerful for software quality assurance. However, the biggest benefit can be gained with the elaborated integration of all three types of tools. Such a deep integration of course requires well-defined interfaces. The interfaces used in this article can be briefly summarized as “command line” for launching test automation and “xUnit” for understanding reports. This article provides  step-by-step instructions to set up such an integration based on common representatives of test management tools (TestLink) and continuous integration tools (Jenkins) with Ranorex as the automation tool.

Resources and Further Reading

Download Links

Bitnami WAMP installer: https://bitnami.com/stack/wamp/installer

TestLink installations (https://bitnami.com/stack/testlink)

TestLink plug-in for Jenkins (https://wiki.jenkins-ci.org/display/JENKINS/TestLink+Plugin)

Jenkins CI (http://jenkins-ci.org/)

Resources

TestLink website (http://www.testlink.org/)

TestLink installation and configuration manual (http://testlink.sourceforge.net/docs/documents/installation.html)

Jenkins TestLink plug-in tutorial (http://tupilabs.com/books/jenkins-testlink-plugin-tutorial/en/book.pdf)

Using TestLink and Jenkins to automate test results logging (http://gazelle.ihe.net/content/use-testlink-and-jenkins-automate-tests-results-logging)

Ranorex Blog Posts

Integrate Ranorex into Any CI Process

Integrating Ranorex Automation in Jenkins CI

Fully Integrate your Ranorex Report with CI Tools Like Jenkins Using xUnit