Let’s be honest: We rarely test the product functionality under load. But how can we be sure our end product works when our customers are using it? As we’ve described in our previous blog post “Combining Automated Functional and Load Testing”, it often makes sense to combine functional and non-functional tests. A functional test, which works fine in idle conditions, might fail when the back-end server is under load. Just like simply stressing a back-end system may not reveal functional issues, which can only be found by an automated functional test. If we want to find those errors that only occur under load, we have to combine automated functional tests and automated load tests.We’re happy to announce that you can now combine Ranorex and NeoLoad tests!

In this blog, we want to show you how you can set up the Ranorex-NeoLoad integration and what you can do with it. But first, let’s quickly cover the basics:

What is NeoLoad?

NeoLoad is an automated load and performance testing tool from Neotys.

NeoLoad offers a full-fledged REST API to either remote control the execution of a NeoLoad test or transmit timing values to NeoLoad. To enable integration with Ranorex, the REST API calls are wrapped with Ranorex functions and packaged into a NuGet package for easy deployment.

What do I need to enable the Ranorex-NeoLoad integration?

• Ranorex license – alternatively, you can download a free 30-day trial
• NeoLoad Professional license – alternatively, you can also use the free license

Now that you’re all set, we want to show you in detail how you can:

1. Set up the Ranorex-NeoLoad integration
2. Use the load testing modules available with the integration
3. Transmit navigation timing to a NeoLoad test
4. Update meta-information in cross-browser tests
5. Upgrade an existing Ranorex project with the Ranorex-NeoLoad NuGet package

Setting up the Ranorex – NeoLoad integration

First, we need to set up the integration:

Add the NuGet package to the Ranorex project

• Right-click on “References” in the Ranorex project view
• Select “Manage Packages…”
• Search for “Ranorex” and add the “Ranorex-NeoLoad integration” package

This will automatically add the necessary libraries to the Ranorex project. The following code modules will now appear in the module browser: 

You can now use the modules, which are included in the NuGet package, freely within the Ranorex test automation project.

Modules included in the Ranorex-NeoLoad NuGet package

The following modules, and their individual variables, are included in the Ranorex-NeoLoad NuGet package:

• NL_ConnectToDesignAPI
• NL_CreateProject
• NL_OpenProject
• NL_CloseProject
• NL_SaveProject
• NL_StartRecording
• NL_StartRecording_Extended
• NL_StopRecording
• NL_StopRecording_Extended
• NL_ConnectToRuntimeAPI
• NL_StartTest
• NL_StopTest
• NL_AddVirtualUsers
• NL_StopVirtualUsers
• NL_ConnectToDataExchangeAPI

For a full complete documentation of each module please refer to NeoLoad module definitions.

Transmit navigation timing data from any browser to NeoLoad

Opening a website is related to a certain latency. This latency depends on various factors, such as the network connection or the browser used. It can be measured with the “Navigation Timing” API, which is offered by all browsers. If you evaluate these timing values, especially when the website is under load, you can localize potential bottlenecks. Eliminating the identified bottlenecks will ultimately improve the user experience.

 The NuGet package offers a mechanism to calculate these timing values and transmit the results to NeoLoad. You can find a more detailed description of the navigation timing here. The timing values are calculated by the Ranorex/NeoLoad Nuget package:

Highlighted in green, you can see the timing values that are calculated by Ranorex and submitted to NeoLoad.

To transmit the timing values, you need to add a new user code action  into the action table in Ranorex Studio. Once the NuGet package is added to the Ranorex project, the additional user code action “SendTimingValues” will appear in your projects user code library.

The “SendNeoLoadTimingValues” action accepts the dom node from you repository and a “transaction name” as an argument. We recommend using the current page as a transaction name in the Ranorex action table. As soon as NeoLoad receives the timing values of this transaction, a tree with the root node containing the Ranorex test suite is automatically created. Another subfolder is automatically created for the respective transaction name. This folder contains the timing values transmitted from Ranorex.

Important:  Please make sure to initialize the module “ConnectToDataExchangeApi” before you use the module “SendNeoLoadTimingValues”. Otherwise, an error is thrown. 

You can drag the data series into the graph board in NeoLoad to visualize it. If you’ve provided meta-information, such as “Hardware”, “Software” or “Location” in the “ConnectToDataExchangeApi”, you can now use this information to filter timing values transmitted from Ranorex.

Update meta-information in cross-browser tests

If you execute the test in multiple browsers, you have to update the filter options in NeoLoad by calling the “ConnectToDataExchangeApi” module again. To do so, bind the data column, which specifies the browsers, with the “Software” argument from the “ConnectToDataExchangeApi” module. You can now compare timing values from different browsers.

Exemplary Ranorex project structure

In the screenshot below you can see an example of how you can use the modules provided in the NuGet package within a Ranorex test project:

As you can see, a connection to the runtime API is established in the global setup section. The login information take the form of global parameters. At the very beginning, “StartNeoLoadTest” starts the NeoLoad test scenario. The following test case is data driven and provides the number of virtual users that will be added to the test. These values are provided in “AddVirtualUsers”. The inner loop is a cross-browser loop. It defines the browsers in which the test will be executed.

Please note: The module “ConnectToDataExchangeApi” can be called multiple times to update the current browser with the filter feature in NeoLoad.

Upgrade an existing Ranorex project with the Ranorex/NeoLoad NuGet package

If you add the NuGet package to an existing Ranorex project, which already contains a Ranorex Object Repository with repository elements, the modules provided by the NuGet package are automatically available in the module browser. In this case, the “SendNeoLoadTimingValues” option won’t be available in the “Dynamic Actions” list for the already existing repository items. Perform the following steps to enable this option: 

1. Open the RanoreXPath editor

2. Switch to “Browser & Results”

3. Drag and drop the root element from the Ranorex Spy to the matching root element in the Ranorex Object Repository.

Now, the “SendNeoLoadTimingValues” will be available in the Dynamic Actions list for the repository root element that describes the website DOM.

Conclusion

In this blog, you’ve learned how you can combine Ranorex and NeoLoad tests. You’ve seen the modules and variables that are available with this integration and how you can transmit timing values to a NeoLoad performance tests. Now, you will be able to validate critical business cases under load conditions to ensure system stability under real usage conditions and identify potential bottle-necks across technology borders.

Further Resources

• Blog Combining Automated Functional and Load Testing: Read this blog to learn more about the benefits of combining functional and load testing.
• Webinar Combining Automated Functional and Load Testing Webinar: Watch this webinar for free and learn why and how to combine automated functional and load tests.
• Blog Ranorex-NeoLoad Webinar – Questions Answered: Get answers to questions asked during the Ranorex-NeoLoad webinar to fully understand how to set up the Ranorex-NeoLoad integration.

Watch the Ranorex-NeoLoad Webinar

Our Ranorex-NeoLoad webinar was a great success and we were delighted to see how many of you are interested in the Ranorex-NeoLoad integration. In this webinar, we’ve showed you in detail why it makes sense to combine automated functional and load tests, how you can set up the Ranorex-NeoLoad integration, and what you can do with it. For those of you who’ve missed the webinar, we’ve recorded it and you can view it anytime here: Ranorex-NeoLoad webinar.

Thank you again for all the questions you’ve asked during this webinar. As there wasn’t enough time to answer all questions during the webinar, we’ve taken this opportunity to answer a few of them, that we believe are interesting to all of you, here:

Q1: Can I test my desktop app, which is connected to a server, with NeoLoad and Ranorex?

Yes, that’s possible. If the communication is based on the HTTP protocol, the NeoLoad recorder captures the traffic utilizing the HTTP transport layer. If the communication is not based on HTTP, you can use custom actions. Ranorex is then able to determine performance values of the desktop app and transmit these values back to NeoLoad.

Q2: I have already created Ranorex web tests. Can I transform functional test sequences from these tests into NeoLoad load tests?

Yes. Simply start a NeoLoad recording manually, and then start the Ranorex test. NeoLoad will record all steps that are being carried out by Ranorex, so that the Ranorex test sequence is available in NeoLoad. You can now put a load on your server with NeoLoad and see if your functional test sequence is still successful when under stress.

Q3: Where should I install Ranorex and NeoLoad? Can everything run on the same machine or should we use separate systems?

In theory, you can install the NeoLoad controller on the same machine as Ranorex is installed on, but that’s not recommended. Load generation can cause a high CPU usage and the controller needs a lot of memory. That’s why it makes sense to install both the NeoLoad controller as well as the load generator on a different machine than Ranorex is installed on. The modules, which are included in the Ranorex-NeoLoad  NuGet package, enable communication between Ranorex and Neoload regardless of where Ranorex and Neoload are installed. However, if NeoLoad triggers a Ranorex test, a NeoLoad load generator must be installed on the same machine as the Ranorex test runs on.

Q4: Are virtual users in NeoLoad carrying out the same actions as the Ranorex automated test?

This completely depends on your setup. The “test sequences” are not linked together in any way. You’re free to set up the same sequence or use a different one.

Q5: When should I start my Ranorex test with NeoLoad, and when does it make sense to start my NeoLoad test with Ranorex?

If you run a functional test session and want to make sure that individual functional test sequences still succeed when the system is under load, it makes sense to trigger and control the NeoLoad tests from Ranorex. The other way round, if you’re running load testing scenarios and want to know if key functional uses cases still work, it makes more sense to trigger the respective Ranorex tests from NeoLoad.

Q6: Is there a limit on virtual users, which Ranorex can add to a NeoLoad test?

The number of virtual users is not limited by Ranorex in any way. You can add as many users as your NeoLoad license provides.

Q7: Do we need a Ranorex Runtime License for each virtual user in NeoLoad?

No. You can add as many users as provided in your NeoLoad license with a single Ranorex Runtime License. The Ranorex license does not influence the NeoLoad license in any way and vice versa.

Q8: Is there a free version of Ranorex and NeoLoad so I can try the integration?

You can download a free Ranorex trial here: Free full-featured 30-day Ranorex trial. A free version of NeoLoad can be downloaded using this link.

Q9: Is it possible to use Ranorex tests to create load instead of using NeoLoad?

You could also run multiple Ranorex tests in parallel on multiple virtual machines. However, Ranorex is only able to run a single test per machine, as it is a functional test automation tool that automates real mouse and keyboard events. That’s why it makes sense to use NeoLoad to create load.

Q10: What are the system requirements for Ranorex and NeoLoad to enable this integration?

You can find the system requirements for Ranorex here: Ranorex system requirements. You can find the NeoLoad system requirements on this page: NeoLoad system requirements.

Q11: Can we trigger Ranorex and NeoLoad tests from build or CI systems like Jenkins, TeamCity, TFS, etc.?

Yes, that’s no problem at all. Ranorex creates standalone executable files, which carry out the test. These files have to get triggered by the CI system. You can find instructions on how to integrate Ranorex tests in your CI system in our blog “Integrate Automated Testing into Any Continuous Integration Process“. NeoLoad also offers command line execution of load tests. You simply have to start the ‘NeoLoadCmd.exe’ file, as described in detail on this page.

Further Resources:

• Blog Combining Automated Functional and Load Testing: Learn more about the benefits of combining functional and load testing.
• Blog How to Combine Ranorex and NeoLoad Tests: Find instructions on how to set up the Ranorex-NeoLoad integration.

Agile has changed the way we approach development and testing. At least it should. Release cycles are shorter, requirements change rapidly and quality standards are higher than they’ve ever been. To have the slightest chance at scoring a medal in the agile testing games, we’ve got to understand the rules of the game and get the right tools. So let’s get at it together.

I am trying to break down the requirements agile testing should meet on five basic principles. Based on these, I’ll show you the benefits we’ve experienced ourselves when using Ranorex for agile testing and how your team will also benefit from using Ranorex for agile test automation.

Fast feedback

Best case scenario: You test the full range of functionality for every single code change. When performing continuous testing, it’s essential that the whole development team gets immediate feedback on the executed tests. Therefore, an easy to access and easy to read reporting mechanism is indispensable. This is where our test reports step in. You can fully integrate the lightweight test executables of our test suites into any continuous integration systems and share the resulting easily understandable report files with the whole team to ensure a high level of transparency. Not fast enough? You can also get live feedback during test execution: Simply open the test report while the test still runs to see if any test cases have failed so far. The earlier you know, the faster you can react.

High level of automation

Fact is that manual testing is slow, labor intensive, inconsistent and buggy. If you want to rapidly respond to changing requirements and constant code changes, a high-level of automation is absolutely essential. Next to basic unit tests, acceptance tests and integration tests are highly important to test the full range of functionality. To reduce time and increase quality, test automation has to be an integral part of the project from the beginning. You can use Ranorex Studio to automate your entire range of UI tests.

Low overhead

Sometimes, it make sense to create a simple throwaway test, which is used only in one specific sprint. In these cases, there’s neither time nor resources for a big setup. Using Ranorex Studio, you can easily automate such tests and integrate them in your existing test environment. The lightweight test automation projects of Ranorex result in executable files, which you can directly trigger from command line. The executable files inform you if the test has failed or succeeded. In addition a report file can provide detailed information about the test run. Perfect conditions to integrate into any continuous integration process.

Termination of testing roles

In an agile software development process, the whole team is responsible for quality. The borders between the traditional understanding of testers and developers blur. So, typical testers should be able to write unit tests or simple integration tests. In contrast, developers should also record UI tests. Using the broad set of tools ideal for all skill sets, you can set ideal conditions for developers and testers to work together on projects. Pick and choose – it’s up to you: Easily create script-free automated tests using the Ranorex Recorder, or quickly create and edit tests entirely in C# or VB.Net.

Termination of testing phase

Finally in agile iteration cycles, there’s typically no time for a sequential processing of all testing levels (unit tests, integration tests, system tests, acceptance tests…). Therefore test automation has to be an integral part of the project from the beginning. As Ranorex Studio does not rely on dependencies, you can create test scenarios at any time. As soon as you know how the UI will look like, you can implement an automated test and refine the detailed paths of the repository elements as you go along.

As a result we’ve seen, agile testing is based on a high level of test automation. To set the ideal conditions for your team, make sure to get a tool, or a set of tools, that enable you to directly integrate automated testing in your continuous delivery pipeline, ensure a high level of transparency and enable collaboration within your team.

Do you want to get a head start to GUI testing and learn more about the major GUI testing techniques and types? Go ahead and check out our comprehensive GUI testing guide now!

There’s no denying the importance of Selenium when it comes to web browser automation. While its many automation benefits are obvious, there are common challenges both testers and developers alike encounter when using Selenium. In this blog post, I want to address five common challenges and show you how you can solve them with the Selenium WebDriver integration in Ranorex Studio.

Identifying dynamic content

The challenge

It can be tricky to identify content with dynamically generated attributes using Selenium. Dynamic content is based on dynamic identifiers (IDs). These IDs are generated anew every time an element is displayed, which makes it difficult to address the web element based on this attribute. For example, the IDs in web applications based on the “Yahoo User Interface Library” look something like “yui_3_8_1_1_13679224741219_543”. This ID changes every time the webpage reloads. In this case, you cannot use the ID locator to identify the element.

How to solve it

Ranorex comes with a set of RanoreXPath weight rules that automatically decide which attribute to use to identify objects based on particular web libraries. If you want to resolve a specific problem when identifying web elements, you can add a rule to this set. Please have a look at the blog post “Automated Testing and Dynamic IDs” for detailed instructions. Once added to the rule set, you can create your script-free or code-based test scenarios and run them on your WebDriver endpoints. As a result, the object repository will be automatically filled with robust repository items.

Dealing with timing issues

The challenge

Another challenge, especially when testing dynamic web applications, is handling timing issues – for example, when a query takes longer to provide the desired output. In Selenium, you have to manually implement a wait mechanism in code to overcome this issue.

How to solve it

Ranorex automatically creates search time-outs for each item in the object repository, providing a built-in safety net for possible timing issues. You can edit the search time-out of a repository item in Ranorex Studio (context menu item Properties).  In addition to the automatically added search time-outs, you can explicitly wait for a specific element to appear or disappear using a Wait For Exist or a Wait For NotExist action in the actions table. You can get an overview of all possible actions as well as detailed information about specific actions in the section Types of Action Items in our user guide.

Maintaining web elements

The challenge

Test maintenance is dreary and, unfortunately, unavoidable. Especially in complex test scenarios, it can be difficult to maintain web elements addressed in your automated tests. When using a pure Selenium WebDriver implementation to automate your test scenarios, the same web element may be used multiple times. If this element changes, you then have to alter every occurrence of it manually. Even if you have used the page object pattern to manage web elements, you have to find the element that has changed and fix it in code.

How to solve it

In Ranorex, you can use the central object repository to manage your web elements. Every time an element in your application under test changes, you only have to edit it once and all changes will be automatically applied to all occurrences of this web element. To do so, select the desired element in the repository and open its RanoreXPath. Simply re-track the element using Ranorex Spy to successfully identify the element for future test runs.

Implementing data-driven testing

The challenge

Selenium WebDriver doesn’t have a built-in data-driven testing mechanism. This means you have to manually connect your automated tests to external data sources, read the data out of these sources and execute your test scenario with the added data if you want to data drive your tests.

How to solve it

Data-driven testing is already an integral aspect of Ranorex. Without any additional preparations, you can choose between several types of data connectors (simple data table, CSV file, SQL database and Excel file). Simply use the data from these external sources, to automatically execute your test cases with different data sets. You can find detailed instructions on how to perform data-driven testing in the dedicated user guide chapter.

Reporting

The challenge

An important aspect of any test automation environment is getting a detailed and easily understandable report for each test execution. When you use Selenium WebDriver, there are several possibilities to achieve a reporting mechanism. All of which, however, have to be implemented in code using third-party integrations.

How to solve it

Using the Ranorex Selenium WebDriver integration, you don’t have to worry about reporting. When you execute a website test on a WebDriver endpoint, a detailed Ranorex report will be automatically generated. This report provides you with a comprehensive overview of the entire test execution flow. As it is a JUnit compatible report, you can easily integrate it into any CI process.

As you can see, the matchless Selenium WebDriver integration in Ranorex Studio 7 allows you to finally use the best of both frameworks to your advantage and address common Selenium WebDriver pain points.

When you want to run automated tests using different configurations, you will very likely use virtual machines. However, as with everything, you can run into problems. In this article, I’d like to demonstrate problems you may encounter when executing automated regression tests in CI environments on virtual machines that use different configurations and will show, how you can easily resolve these problems using Ranorex Remote.

• The starting situation (or rather, the problem)
• Solution 1: Reconstruct the error state manually
• Solution 2: Trigger Test Execution on CI server
• Solution 3: Execute Automation Steps Locally on VM
• Solution 4: Execute Automation Steps on the VM Using Ranorex Remote

Let me quickly set the scene:

The starting situation (or rather, the problem)

Let’s say you’re working in quality assurance in an agile environment. If so, you have likely encountered the problem of tests failing on specific virtual machines – and only on these machines. Locally – on your test-creation machine – the tests run smoothly without any error.

How can this happen?

Well, this could be due to several reasons, such as an unexpected pop up, like an update window, or a browser window has not been closed as expected on the virtual machine. Or maybe the system under test behaves differently than the local machine, or the test fails because the location-specific system settings, such the system language, differ from the one on the local machine.

So, how can you handle this error behavior?

Solution 1: Reconstruct state of error manually

The first approach that may come to your mind, is to manually reconstruct the preconditions leading to the error. This means that you have to go through the test report, locate the error and investigate the preceding steps, leading you to the point the error occurs. Once you’ve uncovered how the error can be reproduced, you can connect to the remote machine and repeat all steps necessary to exactly reconstruct the failure state.

This seems to be a legitimate approach if the complexity of the whole test workflow is limited. But the more steps you have to perform manually to reproduce the exact system state, the more time-consuming this procedure gets – and the less it makes sense to pursue this solution.

So, I think this approach is not practical in most use cases.

Solution 2: Trigger Test Execution on CI server

In the next approach, it seems only natural to replace the manual steps mentioned in the previous solution with automated steps.
The automated steps needed to achieve the actual state of your SUT when the error occurred already exist in your test automation project. To replace the manual tests, open the test execution build step on your CI system, and alter the command line arguments to only execute the failed test case. If you have a data connector bound to your test case, you should also update the data range, to reduce the test execution runtime to a minimum. For detailed information about command line arguments, please refer to the user guide section “Running Tests without Ranorex Studio”.

When following this approach, you can only run one test case. This means that you will not be able to run global setup and teardown sections or disable modules, etc., which may be necessary to restore the desired system state.

To overcome this restriction:

• add a new, or update an existing, run configuration, and update, if needed, the data range of your data connectors to reach the desired system state,
• commit your changes to the version control system,
• and trigger the test execution with the updated run configuration on the CI system.

This approach is problematic, as committing unnecessary changes to the version control system will mess up the revision history.

A non-transparent and complex version history is – especially when working together with a development team – not desirable.

Solution 3: Execute Automation Steps Locally on VM

Committing changes in the run configuration to the version control system is not required, because of the reasons mentioned above. So, it would make sense to run the automation locally on the virtual machine without using CI mechanisms at all.
Remotely connect to the VM and see if you can find the automation project on the hard drive. Then, adapt the run configuration and execute the updated automation workflow, so that only the failed test case will be executed.

But wait! In most cases, continuous integration systems clean up the executing machine after build and tests steps have been executed. This means that the probability of finding your test automation project on the virtual machine is rather low.Another possibility is to manually copy the test automation project to the virtual machine, which requires a shared network drive or similar.

There must be an easier and more comfortable way to reconstruct an error on the system under test!

What about using Ranorex Remote to solve your problem?

Solution 4: Execute Automation Steps on the VM Using Ranorex Remote

To do so, simply:

• install a Ranorex Agent on the system under test as explained in our user guide section “Install and Set Up Ranorex Agent”
• update your run configuration to only perform the steps necessary to reach the state the error occurs,
• and execute the test on the previously set up Ranorex Agent from the remote pad as explained in the user guide section “Ranorex Remote – Execute Tests”.

After performing these steps, the system under test should be in exactly the state the error has occurred. Now you can connect to the virtual machine and investigate the failed test case. In my opinion, Ranorex Remote is the fastest and most comfortable way to reproduce failed test cases on virtual machines.

Next to investigating failing test runs, Ranorex Remote is also very helpful during test creation. Simply execute your newly created test scripts on a remote agent. Continue using your machine in the meantime, as the test is executed on the remote machine and your computer will not be blocked. You’ll automatically be notified once your test has been executed and the report is ready. Re-check if everything works as expected, and commit your changes to the version control system.

Your revision history will be happy…

In software engineering, continuous integration (CI) means the continuous application of quality control processes — small units of effort, applied frequently.

In this blog, we’ll show you how to set up a CI job with Hudson/Jenkins that automatically builds and executes Ranorex automated tests and generates test reports for every change committed to a Subversion repository.

• Advantages of Continuous Integration Testing
• Infrastructure
  • Continuous Integration Tools
  • Source Code Management
• Plugins
  • Install Plugins
  • Configure Plugins
• Add a New Job
  • Configure Source Code Management
  • Add Build Steps
  • Add “Run a Ranorex Test Suite” Step
  • Add Post-Build Action
• Run a Job
• Add a Repository Hook
• Conclusion

Advantages of Continuous Integration Testing

Continuous integration has many advantages. When tests fail or bugs emerge, developers can revert the codebase to a bug-free state without wasting time debugging. Developers can detect and fix integration problems continuously—and thus avoid last-minute chaos at release dates. In addition, CI testing provides:

• Early warning of broken/incompatible code
• Early warning of conflicting changes
• Immediate testing of all changes
• Constant availability of a “current” build for testing, demo, or release purposes
• Immediate feedback to developers on the quality, functionality, or system-wide impact of their written code
• Frequent code check-ins push developers to create modular, less complex code

Return to top

Infrastructure

Continuous Integration Tools

Below are download links and installation description for Hudson and Jenkins:

• https://jenkins.io/
• http://hudson-ci.org/

This blog post will use Jenkins as the CI tool. Differences between Jenkins and Hudson are minimal. Note that because Jenkins or the nodes executing the CI jobs normally are started as Windows services, they do not have sufficient rights to start UI-applications. Ensure that Jenkins is not started as a service on the master or on the slave nodes where the Ranorex automation should be triggered.

For the Jenkins master, open the Services tool (which is part of the “Administrative Tools” in the control panel), choose the Jenkins service, stop the service, and set the Startup type to Disabled, as shown below.

Use the following command to start Jenkins manually from the installation folder:

java -jar jenkins.war

After starting Jenkins, use this address to access the web interface: http://localhost:8080/

To configure your Jenkins server, navigate to the Jenkins menu and select Manage Jenkins > Configure System:

Note: It is necessary to have the Ranorex main components—and a valid Ranorex license—installed on each machine where you want to build and execute Ranorex code.

Return to top

Source Code Management

As mentioned before, this example uses a Subversion repository as the base for the continuous integration process. In this sample, we have two solutions in the repository: the application under test and the automated Ranorex tests.

To start the application under test from your test project, simply add a new Run Application action to your action table in Ranorex Studio, which starts the application under test, using a relative path to the repository root:

Return to top

Plugins

Since we want to build the code for each committed change within the SVN repository, we need a Subversion and an MS Build plugin for Jenkins. An additional mail plugin will ensure that an email is sent with each build.

Install Plugins

From the Jenkins menu, choose Manage Jenkins > Manage Plugins. Select the following plugins from the list of available plugins and install them if they are not already installed:

• MSBuild Plugin
• Email Extension Plugin
• Subversion Plugin
• Ranorex Test Execution

Configure Plugins

The installed plugins also need to be configured, following the steps below:

• From the Manage Jenkins menu, choose Configure System. Configure the Extended E-Mail Notification plugin with the recipient(s). Set the default subject and default content as shown below. Including the environment variable $BUILD_LOG with the content will add the whole console output of the build and the test to the generated email.

• Configure the E-mail Notification plugin by setting the SMTP server.
• Navigate to Global Tool Configuration and configure your MSBuild plugin by choosing the msbuild.exe installed on your machine.

Return to top

Add a New Job

Now that the system is configured, you can add a new Jenkins job. This job will update the checked-out files from an SVN repository, build both the application under test and the Ranorex automation project, execute the application under test as well as the automation code and send an email with the report file attached.

Start by creating a new item. Choose Build free-style software project as job type and enter a job name:

Return to top

Configure Source Code Management

The next step is to check out the source of both the application under test and the test automation project. Start with choosing Subversion as the source code management tool. Then, enter the repository containing your application under test as well as your test automation project. Finally, select Use ‘svn update’ as much as possible as the check-out strategy:

With this configuration, the application under test as well as the test automation project will be checked out and updated locally.

Return to top

Add Build Steps

Now that the source code management is configured, you can start with processing the updated files. First, add MSBuild steps for both projects:

Choose the configured MSBuild version and enter the path of the solution file relative to the repository root (the workspace folder of the Jenkins job), for both the automated and the automating project:

Return to top

Add “Run a Ranorex Test Suite” Step

By adding these two build steps, the executables will be built automatically. Now the newly-built application should be tested. This can be accomplished by adding a new “Run a Ranorex test suite” build step that starts the test suite executable.

To see how this works, watch the following short video or follow the steps below:

How to configure the “Run a Ranorex test suite” build step

• Ranorex test suite file: Enter the path to the test suite file (*.rxtst) located in the output folder of your solution.
• Ranorex run configuration: Enter the exact name of the run configuration you want to use. By default, the run configuration currently selected in the test suite is used. If you want to create or edit run configurations, please use Ranorex Studio or the Ranorex Test Suite Runner.
• Ranorex report directory: Specify the directory where your report and corresponding JUnit report file will be saved. If you don’t specify a path, the directory where your test executable is located will be used.
• Ranorex report file name: Specify the file name of the generated report. By default, the file name specified in the test suite settings is used.
• JUnit-compatible report: If checked, Ranorex will create both a JUnit-compatible report and a Ranorex report.
• Compressed copy of Ranorex report: Compresses the report and the associated files into a single archive with the extension .rxzlog. The following additional input fields will appear when this option is enabled:
  • Compressed report directory: Set the directory where your compressed report will be saved. If you don’t specify a path, the directory where your test executable is located will be used.
  • Compressed report file: Set the file name of the compressed report file. By default, the file name specified in the test suite settings is used.
• Global parameters: Create or override values for global parameters set in the test suite. Enter parameters according to the following pattern: “ParameterName=Value”. Separate parameters with semicolons or newlines.
• Command line arguments: Enter additional test suite command line arguments. Please refer to our user guide to get a list of all available command line arguments.

The test suite executable returns “0” on success and “-1” on failure. Based on this return value, Jenkins will mark the build as successful or failed.

Return to top

Add Post-Build Action

If you selected the “JUnit-compatible report” option in the previous step, now you can publish a JUnit test result report to the Jenkins build after the automated tests complete. In this step, you will add a “Publish Junit test result report” post-build action to the build job and define the test report xml, representing your JUnit report file.

In addition, you can send an email regarding the success of the triggered build. This email should include the zipped report file, mentioned before, as an attachment. To do so, add the new post-build action “Editable Email Notification”, choose the report file location defined before as attachment, and add triggers for each job status you want to be informed about. In this sample, an email will be sent if a job has failed or succeeded.

Return to top

Run a Job

Once you have completed these steps and saved your changes, verify that everything works as expected by clicking Build now:

After running the generated job, you should see all finished builds within the build history. Icons indicate the status of the individual builds. If you click on a specific build, you will see the detailed JUnit test results. View the zipped report files of all builds by opening them in the local workspace (“Workspace/Reports”):

As configured in the previous steps, an email will be sent to the specified email address(es), including the console output in the email text as well as the generated zipped report file as an attachment.

Return to top

Add a Repository Hook

Now we can manually trigger a build. Because we are working with Subversion in this example, it would be beneficial to trigger the script for each commit. To do so, add a server-side repository hook, which automatically triggers Jenkins to start a new build for each change committed, as described in the Subversion plugin documentation.

Alternatively, you can activate polling of the source code management system as a build trigger in your Jenkins job configuration. As shown in the following screenshot, you can define the interval, after which the source code management will be invoked (e.g. 5 minutes after every full hour):

Return to top

Conclusion

Following the steps above, you will be able to set up a continuous integration process that includes Ranorex automated tests for the application under test. Each commit will trigger an automated test run. Once the test run has finished, you’ll instantly receive an email with the Ranorex test report.

Note: This blog was originally published in July 2012 and has been revised to reflect recent technical developments.

This article describes how to use the Ranorex Studio IDE and the Ranorex API for test automation in your behavior-driven development (BDD) process. BDD requires a cognitive shift: instead of thinking about testing functions, you are now thinking about behaviors. Following this approach enables seamless collaboration between business stakeholders such as product management, and the QA or development teams. On the business side, the product management team can define feature specifications in natural language without writing a single line of code. On the technology side, the QA team and the development team add automation steps using recording modules and the repository.

Please be aware incorporating BDD with Ranorex Studio is still in the experimental stage. If you require assistance, reach out to our support team or post a question in the user forum.

Tool Stack

The diagram below shows a sample tool stack to create a BDD-based testing framework. Ranorex Studio provides the development environment, while the Ranorex test automation framework provides the automation API. To parse the feature files from natural language to written code, we are using SpecFlow as the BDD interpreter. Finally, MSTest is the unit test provider, which manages and executes the tests.

Process Overview

When following the BDD approach in an agile environment, the entire development process of a user story is divided into a business side and a technology side.

On the business side, we create user stories, which are taken and broken down into features stored in feature files. These features are broken down into scenarios. The features are written down in a natural language following the Gherkin syntax. A basic feature written in Gherkin might look something like this:

Feature: Serve coffee
 In order to earn money
 Customers should be able to
 buy coffee at all times

Scenario: Buy last coffee
 Given there are 1 coffees left in the machine
 And I have deposited 1 dollar
 When I press the coffee button
 Then I should be served a coffee

On the technology side, we need an interpreter to parse the feature files written in natural language (Gherkin syntax) and call test automation code. These Ranorex API calls or Ranorex recordings do the following:

• Automate the system under test
• Verify the expected output
• Returning a corresponding result

Prepare Ranorex Studio for BDD

The SpecFlow Add-in for Ranorex Studio

The SpecFlow add-in provides file templates for feature files, step definition files and event definition files. It also translates the features from Gherkin syntax to C# code. To install the SpecFlow add-in to Ranorex Studio, follow the instructions below:

Download or clone the SpecFlow repo and open the solution in Visual Studio.

If you have installed Ranorex in “C:Program Files (x86)Ranorex 8.0”, the correct references will be added automatically.

If not, open the project “TechTalk.SpecFlow.RanorexIntegration”, remove the following libs from the references:

• ICSharp.Core
• ICSharp.SharpDevelop
• ICSharp.SharpDevelop.Dom

and replace them with the dlls from <Ranorex installation folder>bin.

Next, build the RanorexIntegration add-in.

Extract the built “.sdaddin” file from the “IdeIntegration” folder of the SpecFlow solution to a new folder “TechTalk.SpecFlow.RanorexIntegration“.

Copy the extracted folder to <Ranorex Installation folder>AddinsMisc.

Make your Ranorex Solution BDD ready

First, download the SpecFlow 1.9.0 NuGet package file.

Next, open Ranorex Studio and create a new solution. From the main menu, select Tools > Options. Open the Package management settings, select Package sources, and add a local package source referring to your download folder.

Click OK to close the Options window.

Use the NuGet package manager to add SpecFlow v1.9.0 from your local package source, as follows: Open a solution, then select Project > Manage Packages. Click the Available tab. Use the drop-down to change the source to your Downloads folder. Click the Add button to install SpecFlow. To confirm that it is installed, click the Installed tab. To close the Manage Packages dialog, click Close.

Open the app.config file in Ranorex Studio. In the SpecFlow section, change the unit test provider to fit your needs. In our approach, we will use MSTest (<unitTestProvider name=”MsTest” />), but you can use other providers, such as NUnit.

For a full list of available unit test providers, refer to the SpecFlow wiki on GitHub.

When using MSTest you must also add the Microsoft.VisualStudio.QualityTools.UnitTestFramework.dll to the references of your Ranorex Studio project (“<Microsoft Visual Studio installation folder>Common7IDEPublicAssembliesMicrosoft.VisualStudio.QualityTools.UnitTestFramework.dll”).

Get started with BDD

Now that your Ranorex Solution is BDD-ready, you can begin defining features in SpecFlow and creating automated tests for them in Ranorex. You can see how to do this in the screencast “BDD and Ranorex,” or you can read the instructions below, whichever you prefer.

Define a feature from the business side

In SpecFlow, create a new feature file. To do so, right-click on the project, then choose Add > New Item > Category: SpecFlow > SpecFlow Feature.

Describe the behavior of your system using human-readable syntax in Gherkin. For Example:

Feature: Calculator
 In order to avoid silly mistakes
 As a math idiot
 I want to be told the square root of a number

Scenario: Square root of a Number
 Given I click 3 on calculator
 When I press sqrt
 Then the result should be 9 on the screen

Save the file. Ranorex Studio will create a <name>.feature.cs file containing the automatically-generated code that will execute the test scenarios using the configured test provider.

If the feature file is not created in Ranorex Studio, you have to set the “Custom tool” in the files properties to “SpecFlowFileGenerator” to make Ranorex Studio create the code file.

Create step and event definitions for the technology side

Create a new step definition file. To do so, right click on the project and choose Add > New Item > Category: SpecFlow > SpecFlow Step Definition.

Now, create methods for all “Given,” “When,” and “Then” actions described in the scenario description. Add “Given”, “When” and “Then” attributes with values matching the scenario steps defined in the feature file. To do so, follow the samples provided by the template from which the new step definition file has been generated.

     public class StepDefinition1
     {
         [Given("I have entered (.*) into the calculator")]
         public void GivenIHaveEnteredSomethingIntoTheCalculator(int number)
         {
             ScenarioContext.Current.Pending();
         }

         [When("I press add")]
         public void WhenIPressAdd()
         {
             ScenarioContext.Current.Pending();
         }

         [Then("the result should be (.*) on the screen")]
         public void ThenTheResultShouldBe(int result)
         {
             ScenarioContext.Current.Pending();
         }
     }

SpecFlow searches for the step definition text in the step definition files that matches the text in the scenario step.

Now you can implement the automated step, either by creating recording modules or by manually implementing the steps using the Ranorex automation API.

Your implementation might look something like this:

     public class sqrt
     {
         [Given("I have entered (.*) into the calculator")]
         public void GivenIHaveEnteredSomethingIntoTheCalculator(int number)
         {
             ClickNumButton.repo.numButton = number.ToString();
             ClickNumButton.Start();
         }

         [When("I press the square root button")]
         public void WhenIPressSqrt()
         {
             PressSqrt.Start();
         }

         [Then("the result should be (.*) on the screen")]
         public void ThenTheResultShouldBe(int result)
         {
             ValidateResult.Instance.result = result.ToString();
             ValidateResult.Start();
         }
     }

You can create additional steps to be executed before and after a scenario, such as performing setup and tear down tasks, including starting and closing the system under test.
These steps can be defined in an event definition file and might look something like this:

         [BeforeScenario]
         public void BeforeScenario()
         {
             StartSUT.Start();
         }

         [AfterScenario]
         public void AfterScenario()
         {
             CloseSUT.Start();
         }

Make the project ready for the unit test provider

After implementing the automation steps bound to the feature description, you can build your project as a library so that it can be passed to MSTest. To do so, open the project properties and set the output type of the project to “class library”. In addition, copy all Ranorex runtime DLLs to the output folder.

Run a BDD test using MSTest

Because this solution uses MSTest as the unit test provider, we must use MSTest.exe to execute our tests. Refer to the articles Run automated tests from the command line using MSTest and MSTest.exe command-line options for a detailed description of the MSTest command line tool.

Your command line call should look something like this:

MSTest.exe /testcontainer:<solution>.dll /noisolation

Now the steps defined in the feature file will be executed and the result will be stored in a test result folder. Open the result of your test run for detailed information about the performed steps.

Conclusion

The preparation of Ranorex for BDD is complete. From this point forward, your business team can define user stories in a natural language following the Gherkin syntax, and your development and QA team can implement the automation steps in the step definition files.

Time is money, particularly in IT. Every bug, every missed date, every slow-moving process takes time away from getting software out the door and affects the bottom line. Hence, organizations are always looking for ways to improve efficiency: to do more in less time.

DevOps has proven key to creating the efficiencies needed to stay competitive in an increasingly demanding business environment. History shows that following the fundamental principles of DevOps — cross-functional teamwork, automation wherever possible, fearless communication, and treating infrastructure as code — works! However, achieving a sound DevOps practice doesn’t happen by magic or overnight. Planning, commitment, and hard work are required. This is especially true when it comes to implementing test automation in a DevOps pipeline. It’s not a one size fits all undertaking. Rather, effective implementation requires that companies pay attention to 3 essential considerations. These considerations are:

1. Build flexibility into the DevOps pipeline.
2. Segment the testing process across the pipeline.
3. Understand that not all testing can be automated.

Allow me to elaborate.

Build flexibility into the DevOps pipeline

A typical DevOps deployment pipeline is divided into stages, with each stage having a distinct purpose, as shown below:

The purpose of a DevOps pipeline

The image above shows just one possible DevOps pipeline. In practice, the number, name, and implementation of stages in a DevOps pipeline can vary. Some organizations have three stages while others have six or more stages. Regardless of the number or names of the stages, the important thing is that these are consistent and that the purpose of each stage is well understood and supported on a company-wide basis.

The purpose of a DevOps deployment pipeline is to create stages through which code moves and is inspected and tested. Testing becomes more stringent at each stage and thus reliability and quality increases.

During the first stage — Development — the rate of change in the code is high. New features are added and bugs are fixed. Change can happen on an hourly basis. By the time the application escalates to the final Release stage, the rate of change diminishes to a near standstill as the code becomes more thoroughly tested. By the Release stage,  the code is hardened and ready to be published to the public.

Each stage in the DevOps pipeline has a purpose. Testing is performed according to the scope and purpose of each stage in the process. The pipeline is predictable. Code moves from one stage to another in sequence. That’s how it’s supposed to go … until it doesn’t.

Why flexibility is critical

No matter how well defined a company’s DevOps pipeline is, there will be exceptions that need to be accommodated. Usually, the exception is a hotfix. A hotfix is a piece of mission critical code that needs to be released as soon as possible, almost immediately.

Hotfixes happen all the time. When they do, some companies panic, circumvent the pipeline and send code right from development to release while hoping all will work out for the best. Companies more adept with DevOps will have the benefit of foresight and understand that there are times when the DevOps pipeline will need to accommodate episodes of code escalation outside of the usual deployment process. These companies build flexibility into their DevOps pipelines. They actually have a process by which to fast track code to release while not losing the benefits that a formal pipeline provides.

Such fast-tracking might mean creating a “side stage” in which the hotfix code is subjected to intense, controlled testing outside of the usual deployment sequence. Then, after the hotfix is released, a flexible deployment process will downstream merge the fast-tracked code back into the usual stages in a controlled manner.

Controlled, downstream merging in response to a hotfix is just one technique for providing flexibility in the DevOps pipeline. Each company will have its own techniques for addressing change that falls outside the usual deployment process. The trick is to be able to implement flexibility into the DevOps deployment process in ways that are controlled and safe.

When it comes to creating a viable DevOps deployment pipeline that is built to last, a flexible system will go the distance better than one that is rigid.

Segment the testing process across the pipeline

An important part of the DevOps pipeline is the ability to test code automatically once the code enters a particular stage of the deployment process. It just makes sense. Automated testing provides more efficiency than testing manually. However, efficiency can be compromised when automated testing is conducted in an arbitrary manner. Remember that each stage in the DevOps pipeline has a purpose. Conducting tests that fall outside the purpose of the stage is not a wise use of time.

The purpose of testing in the Development stage is to ensure that the application works according to expectation at the source code level. Typically such testing is accomplished by exercising the unit tests that ship along with the source code.

Component and functional testing take place when the code is built into components and the components interact with each other. Such testing usually takes more time than unit testing, despite being an automated undertaking. Component and functional tests are typically executed in the Staging part of the DevOps deployment pipeline.

Performance testing, which can involve creating millions of virtual users to go up against the application in a variety of network configurations can be conducted at the later part of Staging or in User Acceptance. Performance testing takes more time than component and functional testing. And then there’s Security testing, which can take just as long as Performance testing.

What’s important to understand is that just because you can conduct all tests at any stage of deployment, it doesn’t necessarily follow that you should. Tests take time. Running all tests, in every deployment stage can create time-consuming redundancy. The trick is to use testing time wisely to achieve the highest degree of quality possible. Thus, test planners will do well to work with DevOps personnel to determine the best places in the DevOps pipeline to conduct testing according to the purpose of the pipeline and the purpose of the testing. Thorough testing is critical to a successful DevOps pipeline. So is testing at the right time, in the right place. Segmenting automated testing properly along the DevOps pipeline will go a long way toward implementing a deployment process that is both efficient and reliable.

Understand that not all testing can be automated

Automated testing is essential in the philosophy and practice of DevOps. But, it’s important to realize that not all testing can be automated, particularly as you move into the later stages of the DevOps pipeline where the use cases can become more complex. Some tests really do need to be implemented by humans in real time. Performance and security testing scenarios that are notable use cases for manual testing usually involve a one-off situation that has a variety of special user personas or edge case operations. For example, performance testing under a special promotion scenario for an e-commerce site or doing security testing for a missile launch. These situations don’t happen every day and when they do happen, conditions tend to be unique to the testing time. In such cases, the time it takes to plan and do the manual testing might indeed be less than the time it takes to automate the process.

Granted, most testing can be automated and should be. But, companies will do well to understand that there is a time and a place for manual testing. They should plan on it. This means having the expertise on hand to design and implement the tests. Without proper planning, accommodating manual testing can incur an excessive cost. Planning for the use of manual testing will save time and money.

Putting it all together

Effective DevOps pipelines are not only flexible enough to support the special, episodic demands of the organization but are also structured enough to produce consistently reliable operations. Also, those organizations that have the foresight to segment their automated testing throughout the deployment pipeline, while also reserving the option to conduct manual testing when necessary, will experience greater value from their testing efforts.

The DevOps pipeline is an essential part of the DevOps way of life. It’s the place where all members of the team gather to create and deploy the software that makes the company move forward. And, as goes the pipeline, so goes the team. Paying attention to the considerations presented above will result in more efficient behavior all around.

Modern UI testing is more than firing up an application’s website and then entering data and recording results. Today, a distributed application will be tested in a variety of release stages — from a developer’s local machine, onto a number of progressive external release environments. Such is the nature of Continuous Integration/Continuous Deployment (CI/CD).

Making it so automated UI testing can run smoothly in a CI/CD process that traverses a variety of build environments, is a key requirement of the software development lifecycle in the modern enterprise.

This three-part series describes how to deploy and run GUI automation against different release environments. This first part will cover using Jenkins to deploy web application source code from GitHub to a build server and then perform a GUI test on the application using Ranorex’s web testing features.

The second part will show how to use Jenkins to deploy the code to a staging environment in which the web code runs in a container. We’ll use Ranorex’s web testing features to execute tests against the container.

Finally, in Part 3, we’ll deploy the containerized web application to the cloud and run Ranorex web tests against the application running under Kubernetes cloud-native environment.

Figure 1 below shows the deployment architectures for all three stages.

Let’s get started with Part 1, deploying and building source code in a Development environment and then running Ranorex web tests against the running web application in the environment.

Understanding the Development Runway Architecture

The Development runway is the deployment path for code that is checked into a project’s development branch on GitHub. The purpose of the Development runway is to provide an application environment in which code from all the developers is unified into a single code base that can be reviewed and tested.

The way the Development build process works is that once the developer checks code into the GitHub repository’s development branch from his her local machine, the Jenkins build server takes the code from GitHub and sends it to a Windows slave server where the sample application is built, started and tested using a Ranorex web test. (See Figure 2.)

The reason the code needs to run on a Windows server is due to the fact that Ranorex is a Windows product. The Demonstration Application is written in Node.js and is agnostic in terms of Operating System. But, we need to run on Windows to perform the UI testing using Ranorex.

The following sections show you how to create the Windows Slave server bound to a Jenkins server running on Linux. Also, you’ll learn how to create the Jenkins job that will get the code from GitHub, test it using Ranorex and then publish the test results in the Jenkins UI.

Understanding the Demonstration Application

The code we are going to deploy and test is a trivial Node.js application that is stored on GitHub. We’re going to conduct a Ranorex Web UI Test that exercises the application’s login feature, The UI test enters username and password data. Then it clicks the login button and verifies that the expected greeting message appears. Figure 3 below shows the demo application that’s to be tested and the Ranorex UI test that will be conducted.

The Ranorex test described above will execute on a Windows slave server bound to Jenkins.

Configuration Prerequisites

Before we go into the steps required to create the Jenkins project to run Ranorex UI test on a Windows server, we need to make sure that the Jenkins master server has the necessary plugins installed and the Windows server has the applications needed.

Jenkins Configuration Prerequisites

The Jenkins server requires that MSBuild Plugin is installed in order to execute the build of the Ranorex .NET test solution.

The Jenkins server also requires that the Ranorex Plugin is installed. This plugin will run the test on the Windows server.

The Windows Configuration Prerequisites

The Windows Server requires that the following applications are installed directly on the host system:

• Java
• A licensed copy of Ranorex Studio (or the 30-day free trial version)
• The Chrome browser
• NodeJS
• forever-win (run, npm i forever-win -g after Node.js is installed)

Configuring the Jenkins Server with a Windows Slave

In order to run a Ranorex test under the Jenkins server, we need to connect a Windows server as a slave to the Jenkins master. Once connected, Jenkins will execute the project’s build tasks on the Windows server.

Conceptually, the way the Jenkins master to Windows slave works is that we’ll configure Jenkins to be aware of the Windows server and know the location of particulars in the Windows server file system. Also, we’ll make it so the Windows slave facilitates the connection back to the Jenkins server using credential information provided by the Jenkins master server. Configuring the Windows slave back to the Jenkins master occurs on the Windows Server. Figure 4 provides a visual description of the Jenkins master to Windows Server binding interaction.

Before we can have Jenkins connect to the Windows Server, we must enable Windows to connect to Jenkins using Web Launch. This is done under Configure Global Security under the Jenkins server as is described in the next sections.

Enabling Slave Binding Using Java Web Start

From the right side of the Jenkins Server, click on Manage Jenkins to display the Manage Jenkins web page. Click Configure Global Security as shown below in Figure 5.

Once in the Configure Global Security page, go down the page to the section Agents/TCP port for JNLP agents. Select the option Random as shown in Figure 6, below. This action will enable Web Launch to bind the Windows Server to the Jenkins master server.

Now it’s time to connect the Windows server to the Jenkins master.

Creating the Windows Slave

Go back to the Manage Jenkins page and select Manage Nodes as shown in Figure 7, below.

In the Manage Nodes section, select New Node from the left side menu, as shown in Figure 8.

When you select New Node, you will be presented with the dialog shown below in Figure 9.

Give the node a Name in the textbox. In this case. we gave the node the name, Windows Slave.

In the textbox labeled Remote root directory, enter the location on the Windows server where Jenkins will put its workspace folder and files. Make sure the directory exists on the Windows server.

Give the node a label by entering a value in the Labels textbox. In this case, we entered the label, WinBuildServer. The label is very important. We’ll use it to tell Jenkins where to execute the tasks associated with running the Ranorex test on the Windows slave.

Select Launch agent via Java Web Start from the Launch method drop-down.

If you like, you can enter a value in the text box labeled Custom WorkDir path. Jenkins puts its runtime log files in this directory if told to do so.

Save the settings. You’ll be brought back to the Nodes page. Notice that the newly created node, WindowsSlave has a red X on the computer icon. This is because the external slave has not been bound to Jenkins master. (See Figure 10.)

Now it’s time to go to the Windows slave. Go to the Windows machine and bring up Jenkins server web page under a browser on the Windows machine. Go to the Configure Nodes page on Jenkins web page.  (See Figure 11.)

You’ll notice that the Jenkins web page is displaying the information needed to bind to the Jenkins master server. Jenkins is smart enough to know that the Windows slave needs to be configured to the master and show the UI accordingly. You’ll see information about how to access the agent.jar file as well as the text for the connection command for binding to the master.

Now we need to take 4 steps.

Callout 1 in Figure 11 above shows a link to agent.jar. Download the jar file to your local machine and store it in C:\Jenkins\ directory as defined previously when defining the Windows server.

Open an empty text file in your favorite editor and copy the information provided on the Jenkins web page at Callout 2 into a text editor on the Windows machine. Save the text as a .bat file (Callout 3) in the C:\Jenkins\ directory, as shown in Callout 4.

Click on the .bat file to execute the connection command. You’ll see connection binding activity as output in a command window as shown below in Figure 12.

Go back to the Configure Nodes web page on the Jenkins master after the connection command executes. You’ll notice that the external Windows slave is connected and ready for use. (See Figure 13)

Now we’re ready to set up the Jenkins job to run the Ranorex test on the Windows Slave.

Setting Up the Jenkins Job

We’re going to setup the Ranorex test project as a Jenkins Freestyle project. Go to the Jenkins main page and select New Item from the left side menu, as shown below in Figure 14.

You will be presented with the Item Setup Page. Enter the name of the project, in this case, FeelingTrackerDevBuild. The name is arbitrary. You can choose one that better fits your needs.

Click on Freestyle project, then navigate to the bottom of the web page and click the OK button. The project creation process starts. (See Figure 15)

The project configuration page appears as shown below in Figure 16. We’ll add some description text in the Description text area.

Navigate to the bottom of the General tab to the checkbox Restrict, where this project can be run.

Check the checkbox as shown above in Figure 16. Also, in the Label Expression, enter WebBuildServer, which is the label we gave the Windows slave instance we created earlier. (See Figure 9.) Setting the label tells Jenkins to run this build on the Windows Slave.

Next, we need to tell Jenkins the location of the demo application source code on GitHub. (The code on GitHub also contains the files for the Ranorex test.)

Binding to the Development Branch on Git

Navigate to the Source Code Management section of the project page. Select the Git option as shown below in Figure 17.

Enter the URL for the demo application source on GitHub. Also, set the Branch Specifier to */development. Remember, the purpose of the exercise is to execute a Ranorex test according to a particular release stage. In this case, the release stage we’re focusing on is Development. In the Development stage, we’ll run the Ranorex test directly against the application running from the source code downloaded from the development branch in GitHub. Later, as create Jenkins projects to accommodate builds further along in the Staging and UAT release stages, we’ll get the code from the /staging and /master branch on GitHub respectively.

After we let Jenkins know where to find the source code in GitHub, we need to tell Jenkins how to be aware when new source code is available to test.

Setting Up the Build Trigger

The way we let Jenkins know that there is new code has been committed to the development branch in GitHub is by setting a Build Trigger.

Navigate down the project configuration page to the section, Build Triggers as shown in Figure 18. Select Poll SCM and enter H/15 * * * * in the Schedule text area.

Setting the schedule to H/15 * * * * tells Jenkins to check GitHub every 15 minutes to see if there is new code to download and build.

Having told Jenkins how and when to get the source code from GitHub, we now need to build it.

Building the Node Application

Navigate down the Jenkins project setup page to the Build section. (See Figure 19.) From the Add build step dropdown, select Execute Windows batch command. We need to use a Windows batch command because this job will be running on the Windows slave server. This is a very important distinction. Remember, the Jenkins server is running on a Linux machine, but the Ranorex test will execute on a Windows machine. Thus, we need to execute command line instructions that are compatible with Windows.

In this case, we’re going to tell Jenkins to build the Node.js demo app on the Windows server using the command line instruction. In the Command text area enter the text npm install.

Running npm install downloads all the dependency packages from the internet that the demo application requires. Next, we need to compile the Ranorex test.

Building the Ranorex Test Suite

We’re going to build the Ranorex test suite under a new build step. You’ll see an Add build step drop-down button beneath the previous step we built. Click that button and select, Build a Visual Studio project or solution using MSBuild, as shown below in Figure 20.

In the drop-down labeled MSBuild Version, select the name of the MSBuild plugin you installed on the Jenkins server. In this case, the Jenkins server has an MSBuild plugin named ReselbobPCBuild.

In the MSBuild Build File text box, enter the path to the Ranorex solution file (.sln) that will drive the build compilation. In this case, the entire Ranorex build solution is stored in the GitHub project, in the directory \ranorex\feelingTrackerRanorexTest. (See Figure 21)

When Jenkins retrieves the demo application project files from GitHub, it puts them in the Jenkins project workspace. Thus, we set the location of the Ranorex solution file (.sln) relative to the Jenkins workspace. Jenkins will run MSBuild against the solution file. The executable that is the result of the build will be used to fire off the Ranorex test file in an upcoming build step.

Now we need to start the demonstration application.

Starting the Node Application

Add another Windows batch command to the Jenkins build process, as we did previously in Figure 19. This command will start the demonstration app. In the Command textarea enter: forever start app.js (See Figure 22. Remember, having the forever Node package installed on the Windows server is a configuration prerequisite for running this Jenkins project.)

The forever command starts the demo application in a separate process on the Windows server. If we did not run the demo application in a separate process, the demo application’s running web server would block out the rest of the Jenkins build tasks. The Jenkins build would appear to hang. Running the demo application in a separate process allows the Jenkins build to continue.

Now that the demo application is up and running we can conduct the Ranorex test.

Running the Ranorex Test Case

Add a new build step to the Jenkins project. But this time, select Run a Ranorex test suite, from the build steps drop-down menu, as shown below in Figure 23. (This assumes you have installed the Ranorex plugin previously, as required.)

The way the Ranorex plugin works is that you run the .NET test executable compiled earlier against a .rxtst test file. Thus, in the Ranorex test suite file text box, we enter the relative path to the file feelingTrackerRanorexTest.rxtst that was copied into the ..\bin\Debug\ directory of the Ranorex test solution folder by MSBuild during the .NET compilation process.

Enter a report name in the textbox, Ranorex report file name. Also, click the checkbox for JUnit-compatible report, as shown above in Figure 23.

This step will execute the Ranorex test on the Windows server and generate an XML file, BasicReport.rxlog.junit.xml, which describes the results of the Ranorex test. We’ll use this file in a later step.

Cleaning Up After the Build

After the test has run, we need to shut down all instances of Node.js and Chrome running on the Windows server. We need to do this so that subsequent test runs on the Windows server are not subject to preexisting conditions that create errors. For example, trying to start the demo application’s web server will error, if the application is already running.

To shut down all instances of Node.js running on the Windows server, add a new Windows command build step, and in the Command text area, enter taskkill /im node.exe /F, as shown in Figure 24 below.

Next, add another Windows command build step and enter the command, taskkill /im chrome.exe /F, as shown in Figure 25 below. This command will shut down all running instances of Chrome on the Windows slave.

The last thing we need to do is to configure Jenkins to display the Ranorex test results in the Jenkins UI.

Defining Reports as Post-Build Task

You’ll see the Post-build Actions section at the bottom of the Jenkins project page. Click the Add post-build action dropdown button and select Publish JUnit test result report. (See Figure 26)

In the Test reports XML text area, enter the path to the test XML file we described earlier in Figure 23 when configuring the Ranorex test suite build step. Notice that the XML file will be stored in the ..\bin\Debug directory on the Windows slave server. This directory is where Ranorex builds the test .exe and runs the test. Also, make sure that the name of the test report matches the report name you defined when configuring the Ranorex testing step earlier. If the names do not match up, Jenkins will display an error message in the Post-build section reporting that it can’t find the test file.

Setting up the build is complete. Click the Save button at the bottom of this build configuration page. Now it’s time to run the build.

To run the build, click the link Build Now on the left side menu of the Web page of the build project page, as shown below in Figure 27.

Next Steps

In this article, we looked at how to run a Ranorex test in the Development runway of the release process. We ran the tests on Windows slave bound to the master Jenkins server.

The next step in the multi-phase GUI testing process is to go to the Staging phase. In this phase, we’ll create a new Jenkins build project that gets the demonstration application code from the Staging branch of the GitHub repository. The Jenkins build process for this stage will be to run the demo application from within a Docker container and make it so the Ranorex UI tests execute against the running container instance.

Jira is an issue and project tracking solution from Atlassian. This article describes how to integrate Ranorex Studio with Jira, so that Ranorex Studio can automatically create or modify testing issues in Jira. This integration works with both a local Jira server as well as cloud instances, and is possible because Jira offers a REST web service (API description available here).

Use cases:

This integration supports the most essential use cases for a Ranorex Studio/Jira integration, as listed below:

• Create an issue when a test case fails
• Re-open an old issue for a repeated failure
• Close an issue when the related test case succeeds

In addition, this integration provides automatic issue handling, so that you aren’t required to manually copy and assign the Jira issue key to the Ranorex Studio test case each time that an issue is created in Jira.

Finally, the whole integration is customizable and works on both Jira server and Jira cloud installations.

Structure of the integration

So that you have a single point of control, the integration is designed with a central default configuration, where you can specify the starting values necessary for a running integration. These default values can then be modified as necessary throughout the test execution.

In addition to the default configuration, the integration provides a single module that handles the entire integration. Place this module in the teardown section of the test case for which you want to create a Jira issue.

To modify the settings and values in the default configuration, the integration provides a user code library, allowing you to simply use existing user code actions in the action table.

To provide automatic issue handling, the integration uses a Jira custom field that stores information about the test case.

Getting started

Add the integration to you Ranorex Studio project

A NuGet package with predefined Ranorex Studio modules and user code actions is available from the Ranorex NuGet package manager. To add the NuGet package to your Ranorex Studio project, follow these steps:

1. Right-click on your project in the Ranorex Studio project view
2. Select Manage Packages…
3. Search for Ranorex.Jira and add the Ranorex Jira reporting modules package

In the background, the package manager installs the Jira integration package and all dependency packages (e.g. the Atlassian SDK). You can have a look at them when switching to the Installed tab in the Ranorex NuGet package manager dialog. All necessary resources will be added automatically to your test automation project.

After adding the integration package to your project, there will be a new Jira folder available in the module browser view. This folder contains two modules, one for setting up the configuration and one for automatically handling the whole integration magic.

Note: There will also be a legacy folder in the module browser view which contains the modules from the former community-driven project. If you are using the modules from the former community project, we strongly recommend updating to the new implementation, relying on these two modules and additional user code actions.

Add the default configuration to your test suite

To begin, add a setup region to your test suite:

To add the module to the test suite, simply drag and drop the module InitializeDefaultConfiguration from the module browser view to the newly-created setup region.

Open the global parameters tab of the test suite properties by right-clicking the added module and selecting data binding. Click the Auto-create button to create parameters for all module variables, and then click the Auto-bind button to bind the newly created parameters to the variables.

After that, you can start initializing the integration by entering values for the given parameters.

Required variables:

• EnableJiraIntegration: The default value of EnableJiraIntegration is false. Please specify a boolean value (True/False) to enable it. Otherwise, a warning will appear in the report. This parameter is intended to enable the plugin during CI usage and keep it disabled when maintaining or extending the test suite.
• JiraUserName: The username to connect to the Jira server. For a cloud instance, enter the user’s email address.
• JiraPassword: The password for the specified user.
• JiraServerURL: The URL for the Jira server.
• JiraProjectKey: The project key as specified in Jira (e.g. TES).
• JiraIssueType: An issue type, as available in Jira (e.g., Bug).
• StateClosed: The state which will be set when an issue will be closed (e.g., Done).
• StateReopen: The state which will be set when an issue will be reopened (e.g., In Progress).

Special field:

• RxAutomationFieldName This fields enables an automatic issue treatment by Ranorex. When a test fails, Ranorex Studio searches for an existing issue in Jira where the test case name matches the content of this field. If an issue is found, it is reopened. Otherwise, a new issue is created and the test case name is stored in the issue within this field, so that Ranorex can associate the issue with the test case in the future.Important: A custom field with its name matching the value of this variable must be created in Jira manually!

Other variables:

• JiraSummary: Default summary text for the issue.
• JiraDescription: Default description text for the issue.
• JiraLabels: One or more default labels for the issue separated by “;” (e.g., Mobile; USB; Connection).
• jiraIssuePriority: Default value for the issue priority.
• jiraEnvironment: Default value for the environment field.
• jqlQueryToConnectIssues: Default JQL query to find issues.

Enable the integration for a Ranorex Studio test case

After setting up the default configuration, create a test case which adds a Jira issue when it fails. To do so, simply add a teardown region to a test case in your test suite in the same way that you added the setup region previously.

Now add the module AutoHandleJiraIntegration by simply dragging and dropping it from the module browser view to the newly-created teardown region.

Try out the integration

When you execute the test suite, nothing will be sent to Jira because the test case does not fail. In order to test the integration, you can intentionally force a test case to fail. To do so, open the underlying recording module and simply add a log message with the report level Failure.

Run the test suite again, and the report will show a failure in the test case where you’ve added the log message. If you now open the teardown region of the test case, you will see a log message reporting that a new Jira issue has been created, followed by a direct link to the newly-created issue.

Following the link, you will find a newly-created unresolved Jira issue with an attached Ranorex Studio report containing the information about the failing test case.

When re-running the test, the integration will add a new issue.

To overcome this, you can use a custom single line filed in Jira. Let’s call it RxAutomation.

Note: A custom field can be created by a Jira administrator.

Simply configure this custom field in the default configuration, as shown in the image below.

After configuring this RxAutomationFieldname, the name of the test case will automatically be applied as an identifier of the issue. So every time that the auto-handle module triggers an update, the integration checks whether there are one or more issues where the text in the field RxAutomation matches the test case name and updates the issue(s) if necessary.

When you repeat test runs, instead of creating an issue for every failing test case, the Jira integration automatically handles the updates on the existing issue(s).

Allow the test case to succeed by changing the log message from failure to success, and the issue will automatically be updated to the status done, since the error no longer occurs.

Since the basic integration relys on just two modules, that’s all that is necessary!

Advanced Usage

The integration package includes a collection of user code actions in a user code library. These user code actions can easily be added to your recording modules.

Begin by adding a new test case: simply copy the existing one and give it the name TestCaseAdvanced.

Create a new recording module and add a user code action by selecting it from the user code library, as shown in the image below.

There are two classes defined by the integration, described below.

The JiraFieldHelper provides functionality to:

• Add or remove values from fields
• Set values in fields
• Set the values custom fields
• Set a specific issue key
• Set JQL queries

The second class, ValueExtractorHelper, provides actions to:

• Update the description of the issue with all steps performed in the raising test case (including screenshots)
• Update the issues summary tag by appending the error message

For example, if you want to add a new label, simply choose the user code action addNewLabel and define the label as an argument of the action.

Make sure that the recording module is placed before the auto-handle Jira integration module in the test suite.

Starting the test run will result in an added label to the newly-created Jira issue linked to the test case TestCaseAdvanced.

If you are not able to add a custom field to your Jira instance, you can also use the setJiraIssueKey action to define the specific issue you want to update. You can even use a JQL query utilizing the action setJiraJQLQuery to define a set of Jira issues which should be updated.

To give even more information about why the test case has failed, you can add the actions updateSummary and updateDescriptionWithAllStepsMade to a recording.

This will extend the summary field of the Jira issue with the error message that forced the test case to fail, and also add the entire report of the test case to the description of the issue, including all screenshots.

Of course, all of these user code actions can also be utilized directly from code, if you prefer the code-based approach.

To learn more about the Jira integration check out the free on-demand webinar “Mastering Jira Integration“, or visit the project page on GitHub.

Microsoft Team Foundation Server supports your complete application life cycle management (ALM) process, from source control to automated builds and quality assurance. This blog post illustrates how to implement Coded UI tests that execute Ranorex Studio automation code or even entire Ranorex Studio test suites using Microsoft Test Manager.

Infrastructure

First, you need a working infrastructure to run your Coded UI test with integrated Ranorex Studio automation. Please review Microsoft documentation regarding Team Foundation Server (TFS) Microsoft Test Manager, and Coded UI Tests or this getting started blog post.

In the mentioned infrastructure the Coded UI Test will be implemented on the Visual Studio workstation. This Coded UI Test will kick off the Ranorex automation.

You will use this TFS infrastructure on a Visual Studio workstation to implement a Coded UI test, which will execute the Ranorex Studio test automation. This Coded UI test can then be added to a TFS test case as the associated automation process, as described in this article.

Implementing Coded UI Tests to Execute Ranorex Automation

After providing a working infrastructure, you can create your Coded UI Tests with Ranorex Studio test automation.

You must have the Ranorex Studio main components – and at least a valid runtime license – installed on each machine on which you are going to execute Ranorex Studio code.

After creating a new Coded UI test project and adding a new Coded UI test to it, you could record your Coded UI test. Because we are going to execute Ranorex code instead, you can cancel this step.

The new Coded UI test should contain a test method similar to the one shown below:

[TestMethod]
public void CodedUITestMethod1()
{
}

Starting from this point, you can either implement your own Ranorex code within your test methods or execute an entire Ranorex Studio test suite from your test methods.

No matter which approach you are going to follow, it’s necessary to add references to the Ranorex libraries and add a “using”-directive for the Ranorex namespace as described in the “Visual Studio Integration”-section of our user guide.

After adding the references and the using directive, you can simply start writing your Ranorex automation code as usual.

The recommended way of adding Ranorex Studio test automation to a Coded UI test is to execute either a complete Ranorex test suite or specific Ranorex Studio test cases from your test method.

When running the test suite using the TestSuiteRunner class, you must specify a container type that is in the project containing the test suite. To use the Program class as the container type, you must make that class public. Your Program.cs file should look like the following example:

...
namespace MyTest
{
    public class Program
    {
        ...
        ...
    }
}
...

Change the Output type of the generated project to Class Library and build your Ranorex Studio test suite (<F8>).

After doing so, add a reference to the Ranorex Studio test suite project to the Coded UI test project.

Additionally, add the Ranorex Studio test suite file to (rxtst) to your Coded UI test project and set the Copy to Output Directory behavior to Copy always.

To execute the Ranorex Studio test suite, copy the test suite file to the test machine where the test will be executed.

This is done using the DeploymentItem attribute, which identifies the test suite file and the relative directory where the file should be deployed.

[TestMethod]
[DeploymentItem("MyTest.rxtst")]
public void CodedUITestMethod1()
...

It is crucial that the Ranorex.Core.Resolver assembly is initialized before any other Ranorex core functionalities are used. It is this assembly which finds all other Ranorex assemblies at runtime.

Refer to the Ranorex Studio User Guide chapter Visual Studio integration › Write some Ranorex automation code, for instructions on how to initialize the assembly.

...
public void CodedUITestMethod1()
{
    Ranorex.Core.Resolver.AssemblyLoader.Initialize();
    Ranorex.TestingBootstrapper.SetupCore();
...

Now you can run the Ranorex Studio test suite from the Coded UI test method using the following code snippet:

...
public void CodedUITestMethod1()
{
    Ranorex.Core.Resolver.AssemblyLoader.Initialize();
    Ranorex.TestingBootstrapper.SetupCore();
    Ranorex.Keyboard.AbortKey = System.Windows.Forms.Keys.Pause;
    int error = Ranorex.Core.Testing.TestSuiteRunner.Run(typeof(MyTest.Program), "/zr /zrf:report.rxzlog");
    TestContext.AddResultFile("report.rxzlog");
    if (error != 0)
        throw new Ranorex.RanorexException("Test run failed! See report.rxzlog for further details!");}
...

The first line of the test method will set the abort key. This allows you to stop the automation script on the system under test.

The next line will start the test suite with the given command line arguments.

The argument “/zr” will trigger the test suite to generate a zipped report, which makes it easier to attach the report to the test case executing the test method.
Use the argument “/zrf” to specify the name of the zipped report.

Command line arguments allow you to execute a single test case (“/tc” argument) or launch a defined run configuration (“/rc” argument).

You can find a list of all available command line arguments in the  section “Running Tests without Ranorex Studio” in our user guide.

The next line of code attaches the zipped report file to the test results, which allows you to analyze the test run directly from MTM.

The if statement throws an exception if the test suite execution fails, which will make the test case fail, too.

After checking in the test and building the solution, execute the system under test by triggering the corresponding test case via MTM, as described in this article.

Conclusion

Following the steps in this blog post, you can integrate your Ranorex Studio automated tests in an existing Team Foundation Server infrastructure.

With this integration, you will be able to execute Ranorex Studio UI automation using Microsoft Test Manager.