Hundreds of thousands of software applications are scheduled for migration from Python 2 to Python 3. The deadline is approaching fast: Support ends with 2019.

Abundant articles explain the importance of and techniques for this migration from a developer’s standpoint, including How to Port Python 2 Code to Python 3, Conservative Python 3 Porting Guide and official documentation for Porting Python 2 Code to Python 3, as well as several valuable video tutorials and other resources.

But how can testers help ensure a Python 3 migration is successful? Testing’s unique perspective has plenty to contribute.

Planning and preparation

First, recognize and promote the importance of planning and preparation. Programmers have a tendency to see programming as the solution to problems, and they often benefit from reminders to plan more strategically before the coding starts. A testing department can help developers clarify ahead of time such questions as:

• Is the product source migrated to Python 3 still expected to be able to run with Python 2.7 during a transition period? How long is that transition?
• Will the product source migrated to Python 3 be developed as a branch or on the main trunk of development?
• Once the migrated source goes into production, what will it take to revert to sources compatible with Python 2.7, if reversion is decided necessary?
• Do the testing sources need to migrate? How is their schedule coupled to that of the product sources?
• Can the deployment be targeted for a “quiet time” or “null sprint” when no functional changes are expected? Migration while trying to stay on top of other difficulties and priorities is asking for trouble.
• Will acceptance be on the same rhythm as other releases?

That last point deserves more explanation. Suppose the product to be migrated typically deploys a new release roughly weekly. Once deployments pass all automated tests in this hypothetical practice, they’re regarded as provisionally complete for another 24 hours, to give surprises time to surface. Although deployment schedules vary widely — from minutes to months — most organizations at least recognize these elements.

Should the move to Python 3 be on the same schedule as a “normal” deployment? Any rational answer depends on the comprehensiveness of tests. While a collection of unit tests and a few sanity checks at the integration level might be adequate for most releases, migration to Python 3 tends to bring out whole new categories of problems.

Anything to do with Unicode deserves careful inspection. Quite a few projects have thought their migration complete, then discovered problems only as foreign-language users began to exercise the application. Encoding dependencies pop up in places that don’t occur to the naive: CSV readers, pickles, cookie handlers, cachers and more. Defects or incompatibilities in these places usually have straightforward, backward-compatible resolutions, but someone needs to detect and identify them first. To do so for a release already in production is clumsy at best.

Does your team operate with automated tests? If it hasn’t made that step yet, can it install automated steps before the migration? If the product is a purely text-based library or service, maybe automation is straightforward and yields good coverage. On the other hand, if the product is a web, mobile or desktop application, only a specialized test automation tool can reduce the risk involved in a change as large as that from Python 2 to Python 3.

Ranorex, which underwrites this blog, offers a top-flight testing tool, Ranorex Studio; among other features, Ranorex Studio’s recorder generates customizable scripts in Python. Part of your analysis of the migration should be to define and plan adequate test automation well before deployment of the migrated source.

Performance surprises

Another significant risk of migration has to do with performance degradation. Python 3 is mostly faster than Python 2, and it uses less memory for many important situations — when the source is idiomatic. Problems can arise, though, with subtle semantic shifts.

Python 3 often provides iterators in place of corresponding lists from Python 2; range() is an example. Iterators are generally far thriftier with memory but demand more time when re-created repeatedly. Essentially, expect a Python 3 version to be quicker and take up less memory, but perhaps only after a few adjustments.

A few other hazards of migration are well-documented. Python 2 computes 5 / 3 as 1, while Python 3 gives a result of 1.6666666666666667. In Python 3, the way to arrive at 1 for this calculation is with 5 // 3.

Aligning expectations

All these coding pitfalls have relatively straightforward solutions. Each one demands careful scrutiny though, and errors can combine to upend schedules — and it won’t only be coders who are affected. More even than technical tools or programming heroism, it’s important for testing and development departments to align expectations ahead of time.

Python is on its way to becoming the world’s most popular programming language, so the move to Python 3 will affect many such departments. Explicit plans, well in advance of programming changes, give testers time to prepare and practice with all the materials and tools they need to verify a successful migration

When systems were created with compiled, executable files, a lot of the process of building software was relegated to dedicated build engineers. When I worked at Cisco Systems in the 1990s, we had a dedicated engineering team focused entirely on the build tools and creating the final distribution that would be made available to our customers. For most of us on the engineering team, we had little interaction with the build process. It was built, and then we’d download the necessary images to the machines we would test with.

Today, that process has been opened up considerably by tools that allow anyone on the team to create images for testing. More to the point, through the development of continuous integration tools, like Jenkins, it is possible to make small changes to an application and have those changes start a build, test the build and be pushed to a server with minimal interaction.

Continuous integration and continuous delivery (CI/CD) can seem like mystical ideas, but in truth, they are pretty straightforward. At the center of these practices is the concept of “the pipeline.”

What is a CI/CD pipeline?

The CI/CD methodology hinges on the idea that each step in the process of building software takes place after a previous step has finished. To make it possible for software to be deployed quickly and to apply small changes to test or deliver to customers, there are a variety of steps that can be automated and scheduled to run in a sequence. At their heart, CI/CD tools are scheduling tools that execute bits of code at desired times.

It’s in this framework that a software delivery pipeline is possible. Using Jenkins as an example, in its earlier iterations, I could create a variety of what are called “freestyle jobs” that allow for a set of steps to be executed. By chaining a number of freestyle jobs together, I can create a software delivery pipeline that allows me to commit a code change to a branch and then, from that commit or push, start the process of configuring, testing, building and deploying software to a target machine.

What do I need to build a pipeline?

The first thing to realize is that there are some standard processes most pipelines will have in common, but there isn’t a single approach that will work for everyone. The benefit to the pipeline model is that the stages that are most important to the organization can be applied and made the most effective for that given entity.

What works for my environment may not be practical somewhere else, but an example pipeline approach in my environment looks something like this.

The first step is the merge request builder. This is a process where, if I have a branch that has had some development work done and I push changes to the branch, a hook is triggered that makes a comparison to the original trunk. The system recognizes that a change has been made, and that in turn triggers the build process.

The second step in our environment is based on the fact that we use Docker to allow for parallel tests to be run simultaneously. For that to happen, a sequence of steps is run to bring up a Docker master server, which in turn creates a number of Docker slaves. By configuring these slaves, we create the space where the discrete Docker containers can be duplicated and run.

The third step is what we call the “Docker build” phase, where we bring up the master Docker image, create our test environment using the branch we are doing development with, and sett up the application with all the necessary information and test data needed to run the suite of tests.

The fourth step allows us to take all the tests that are tagged for a given sequence and divide them up based on the number of Docker slaves we have configured. This then allows us to create copies of the Docker image and give each copy a unique ID so that the necessary tests can be run on them. This results in several hundred Docker containers being created and running tests. Through this process, Jenkins monitors the tests and looks for failures. If none are seen (meaning all tests pass), then we go to the next step.

The fifth step is to create the actual branch image and resulting artifacts necessary to install the software effectively on a target machine.

The sixth step is to start up a final target machine — in this case, a dedicated system not within a Docker container. Depending on the system, we can perform a new install or an upgrade based on how the target machine is set up and which flags are configured.

The seventh and last step is a cleanup phase that makes sure that all systems are turned off and that we are not running build or test systems needlessly. This is important because we use a cloud-based solution, and leaving those machines running is, effectively, spending money needlessly. Making sure that everything is cleaned up is vital to the process.

In our environment, currently these are all individual projects configured in Jenkins and then passed on to the next project with successful completion. This is best described as a “sequential pipeline,” where the processes are configured with a dependency on an upstream project and an order of operations.

In the latest Jenkins implementations, the notion of a pipeline has been simplified, and with a Jenkinsfile, the individual steps described above can be put into the Jenkinsfile and run as a single process.

Care and feeding of the pipeline

The purpose of CI/CD — and creating a delivery pipeline in the first place — is to help eliminate bottlenecks and busywork so that the primary focus can be placed where it matters the most: creating features, testing features and delivering features to customers.

By scheduling and sequencing these steps, we are able to add features iteratively and replicate the process reliably. More to the point, each new feature can be checked in conjunction with the rest of the system so we can make sure we have not introduced any defects — or, more specifically, that the changes we have made will not cause other sections of the code to fail.

This is a big benefit compared to trying to compile a system with multiple changes at one go. Still, it has a price: maintaining the pipeline and expanding it as necessary. With new features come new tests. With new tests comes the need for planning for capacity so that tests can be run effectively. Scripts need to be maintained to make sure that the resources are available and running when needed. New tests need to be checked in and run in conjunction with the CI/CD tools to make sure they are working effectively. This all takes time, and every part needs to be maintained.

The CI/CD pipeline will depend on a variety of external tools, such as plugins in the Jenkins example, and these need to be evaluated from time to time and updated and tested with the existing system. However, one other aspect of having a CI/CD pipeline is that, once the steps are codified, they can be maintained by a variety of people and are not just left in the realm of the developers.

The process of making and operating a software delivery pipeline may seem complicated at first, but like all pipelines, they are created step by step and section by section. Each organization will have its own requirements and ways in which it will best be configured. Pipelines can be as short or as long as is necessary, so begin by determining what steps you are already performing and see how you can get started. Each pipeline starts with a first section, so if in doubt, start there.

I remember quite vividly when I was confronted with one of the paradoxes of commercial computing. It was around 1983. The high school where I was teaching opened up a computer lab filled with Apple II computers running the command line-based Apple DOS operating system. I was hearing from a number of sources that computers were cool and I should learn how to work one, so I did.

At the time a friend was also getting into computers as a way of life. One day I was over at his place and he was showing off a program he liked. His computer of choice was an IBM PC running MS-DOS. I remember being impressed with the program and told him I might get a copy for myself. He told me I couldn’t. “Why?” I asked.

His response was, “It only runs on MS-DOS.”

My reaction was akin to a child learning that there’s no Santa Claus. At the time all I understood about operating systems was that a computer needed one to work. I really didn’t have any conception that there were operating systems out there other than the Apple DOS that ran on the machines in the computer lab.
Operating systems mattered a lot back then. In fact, they still do, despite decades of effort trying to unify the computing experience using cross-platform languages such as Java, JavaScript, and HTML.

Those of us who embraced virtualization and cloud computing hoped that these technologies would break the hold operating systems have on the computing experience. They really haven’t. While there have been efforts to make operating systems work in a conceptually similar manner, there are differences that need to be understood in order to have the full benefit of the cloud computing experience.

Remote access: A tale of two operating systems

Facilitating remote access is one such example. Linux supports SSH, which is the way most Linux admin access remote Linux servers at the command line. Windows, on the other hand, uses WinRM to facilitate remote access at the command line to Windows machines. This creates a problem when having to move between Linux and Windows targets using a cloud provisioning technology such as Ansible or a CI/CD product such as Jenkins.

If you want to work remotely across operating systems, you need to know how each OS facilitates remote access. Otherwise, you’re stuck in the water.

There are other differences as well. For example, Windows has a system-wide configuration storage and management technology called the Windows Registry, which is left over from the COM days. Microsoft has been trying to minimize the impact of the Registry through the .NET/CLR technology. But the Registry is still used by native Windows applications and .NET applications that use the COM interop, and vice versa.

A plethora of application installation methods

When it comes to installing applications, operating system matters. Linux has a variety of package management systems: yum, apt and dnf. The one you use is particular to the Linux distribution. Mac’s OSX, which is a flavor of Linux, uses homebrew, MacPorts, and pkg and dmg packaging. On the other hand, native Windows uses MSI Windows Installer, and there is an emerging package manager called Chocolatey. Also, for .NET apps, there is the NuGet platform, which facilitates downloading and installing external libraries.

You can’t run yum on a Windows machine. Conversely, you can’t run the MSI installer on Linux. This difference matters a lot when doing automated machine provisioning.

Linux, OSX and Windows will allow you to install a native app by doing nothing more than copying the executable file to disk. However, this method works only under certain conditions, mostly when there are no dependencies on external libraries. When the primary executable binary has a dependency on an external library, then the way the library is known and accessed varies by operating system and, to some degree, application framework.

For example, Windows might use the Registry to locate the dependency. Or it might rely on a library being present in a predefined location in the file system — for instance, by default .NET looks for dependencies in the bin subdirectory of an application should the given library not be present in the General Assembly Cache. (Of course, the base .NET binaries need to be installed beforehand in order for any .NET application to work.)

Linux will also follow file system convention in some cases. Java tries to make the dependency management the same across operating systems; sometimes it works, and sometimes it doesn’t — it depends on the Java Framework in play. For example, the Maven framework is cross-platform, but it has a dependency management system all its own.

Those pesky little command line differences

And then, of course, there are those pesky command line differences. For example, to copy a file in Linux, you use the command cp, while Windows has copy. To get network information in Linux, you use ifconfig. In Windows, it’s ipconfg. Linux has a filtering command, grep, which is bread and butter for Linux power users, but Windows does not support grep. These are but a few examples of many standard command differences.

In terms of the shell itself, Windows has the standard command prompt. To provide more power at the command line, Microsoft introduced a beefed-up shell called PowerShell, which has a learning curve all its own. PowerShell has some similarities to the standard DOS shell but has much more power. For example, PowerShell supports .NET system objects and C# programming at the command line. Linux goes another route: It has the standard shell, sh, as well as beefier shells such as bash, the korn shell and fish.

Two distinct approaches to doing security configuration

The way you do security for users and groups on a machine is different too. Handling users and groups is a lot easier at the command line in Linux, while Windows is a bit more complex.

The users and groups GUI hides a lot of the management complexity in Windows. Working through the GUI is the preferred method of interaction in Windows, even for Windows Server. However, this incurs a lot more overhead in the form of more disk, memory, and CPU required to support the GUI. Server editions of Linux ship command line only, so they are more lightweight. Yes, you can add a GUI to a Linux server, but that’s considered the sign of an amateur.

Putting it all together

Someone who is used to working with Linux in the cloud and then needs to move over to Windows — for example, to support working with Ranorex Studio — might experience a significant learning curve. The same is true for someone moving from Windows to Linux. Consequently, those working in the cloud will do well to have a basic familiarity working with each OS at the command line. Having such cross-platform understanding will save time and heartache if they ever want to work in an unfamiliar environment.

Not only that, but when you set up your environment, take notes. If it is in the cloud, you can script it up with code. The important thing is to make it reproducible, so the next hire, or replacement, won’t have to go through all that work themselves just to get started.

Red, Green, Refactor

First, a bit of context is necessary. “Red, Green, Refactor” is the widely recognized motto of TDD:

•  Red: Create a test and make it fail
•  Green: Minimally modify the implementation to pass the test
•  Refactor: Eliminate duplication and otherwise enhance the style of the implementation

Tutorials and references appropriately focus on the challenges and subtleties of the Green and Refactor steps; these are more time-consuming. The Red step should be straightforward and quick.

Too often, though, programmers entirely skip Red. To do so undercuts the value of Green and Refactor, and therefore all of TDD. Worse, all the possible remedies for the omission have their own problems.

Consider a concrete, simplified example: Imagine a new requirement that passwords be at least nine characters long. Python is an effective pseudocode for this example. First, a programmer might write

def is_long_enough(trial_password, minimum=9):
'''
'trial_password' is a string.

The return value should be True for trial_password of
nine characters or more; otherwise it's False.
'''

Next, the programmer writes a test:

import unittest

class Tests(unittest.TestCase):
def test_length(self):
self.assertTrue(is_long_enough("long-password"))

At this point, the test probably will show something like AssertionError: None is not true. That AssertionError is a good thing — exactly what TDD tells practitioners they need.

A normal TDD sequence might then continue by updating the is_long_enough definition to

def is_long_enough(trial_password, minimum=9):
'''
'trial_password' is a string.

The return value should be True for trial_password of
nine characters or more; otherwise it's False.
'''
return len(trial_password) >= minimum

At this point, the test result goes to OK or Green.

A Red-less example

Imagine a slightly different example now: A programmer begins writing
import unittest

def is_long_enough(trial_password, minimum=9):
'''
'trial_password' is a string.

The return value should be True for trial_password of
nine characters or more; otherwise it's False.
'''
return True


class Tests(unittest.TestCase):
def test_length(self):
self.assertTrue(is_long_enough("long-password"))

The programmer is interrupted. After a delay, the programmer returns, runs the test, sees OK and concludes it’s time to move forward.

This example is so simplified that it’s easy to see the error: Return True is wrong, and certainly no substitute for return len(trial_password) >= 9. As obvious as that seems now in isolation, I can testify that mistakes of this sort happen in real-world commercial shops on a daily basis. Even top-notch programmers have a tendency to skip Red until they’ve experienced its consequences a few times for themselves.

A temptation

Even then, starting with Green is a powerful temptation. Part of the point of routines and checklists and habits is to shift the execution from our “executive brain” to muscle memory. When we pause to judge whether Red is truly necessary in a case at hand, we deviate from our routine, and we are more prone to stumble in one step or another.

And the melancholy reality is that Red always remains on the periphery of our routines, because we don’t share Red with our peers. The most experienced, zealous TDD practitioners check in or commit their Green work products, but rarely anything that goes Red. Many organizations practice a culture, in fact, that explicitly expects all check-ins to pass all tests. That means that Red is a transient never seen by other programmers. There’s no record of the source at the time of Red and no review of the Red test or its details. Red lacks accountability.

In the normal TDD programming world, Red depends on the good intentions and even ethics of individual practitioners, unsupported by any systematic review, measurement or peering. That’s a formula for failure.

And failure is the too-frequent result. In my own observation, programmers at all levels, even at the very top, at least occasionally lose their footing in the vicinity of Red.

A remedy

What’s the solution? I’ve found no entirely satisfying one. As I see it, these are the main possibilities:

• Continue as we are, with Red subordinate to Green and Refactor
• Relax any CI rules to accommodate commits that are supposed to fail — of course, this complexifies the CI
• Redefine Red as an inverted test that must pass

The latter looks something like this: a first commit along the lines of

import unittest

def is_long_enough(trial_password, minimum=9):
'''
'trial_password' is a string.

The return value should be True for trial_password of
nine characters or more; otherwise it's False.
'''
return True


class Tests(unittest.TestCase):
def test_length(self):
self.assertFalse(is_long_enough("long-password"))

followed by a correction to

import unittest

def is_long_enough(trial_password, minimum=9):
'''
'trial_password' is a string.

The return value should be True for trial_password of
nine characters or more; otherwise it's False.
'''
return len(trial_password) >= minimum


class Tests(unittest.TestCase):
def test_length(self):
self.assertTrue(is_long_enough("long-password"))

The advantage here is that both check-ins become part of the permanent history of the source. Both are available for inspection. The change set between them has a strong, clear relation to the Red-Green transition.

It also looks like more work, and it’s correspondingly unpopular with developers. While I’ve experimented with various modifications of the unittest definition to support the Red step better, none of them yet have the polish working teams require.

Mind the Red

Nearly universal practice treats Red in a way that makes it easy to skip, or at least shortcut. This damages the overall reliability of TDD. No comprehensive solution exists; the best we can do in the short term is to be aware of the hazard and plan organization-specific workarounds.

When it comes to software development, building applications as one large monolith is long gone. Instead, teams build software in components. Sometimes the components are objects and have unit tests; often the user interface is the sum of those components. But even with clean interfaces and distinct components, the only way to test it from a customer perspective is as a whole. The web page or application is in many ways still a monolith because it needs to be built and deployed as one unit.

While we now build in components, too many teams still deploy as a monolith. To deploy components they need to be compiled, tested, and versioned independently.

Let’s start with an example.

Take a look at any popular e-commerce site. The website itself will consist of various pieces you are familiar with, such as a login link (that shows your name if you are logged in) and a shopping cart that shows the number of items in your cart. There will be a search button and several widgets, including things like what you have viewed recently, products recommended for you, upcoming sales, close-out deals and so on.

Each of these items is, in effect, a separate web page; the homepage is just a collection of these sub-pages at specific locations. If one of the services is down, the rest can render, and only the savviest of customers will even notice. The product page also assembles itself out of components: product image, product description, reviews, products other customers also purchased, customer questions and answers, and so on.

There is real functionality in these components. On a modern website, the product image may include videos too, along with the ability to select from many images, change the color of clothing, or mouse over for an expanded image.

Instead of one single build-and-deploy monolith, we can break the website up into a dozen little websites. Each can be built, tested and deployed separately. This reduces the build time for each component from hours to minutes, or less. Large e-commerce sites with over a hundred teams need to do this, but even the smallest of applications can benefit from it, too.

The change is the versioning of those components. Here’s one way to do it.

Building and Testing a Component-Based Application

The answer is in managing different versions at the application or micro-service layer — that is to say, the largest deployment package group that will be deployed together as a single unit. Version control systems like Git, Subversion and even Microsoft’s Team Foundation Server provide mechanisms to group builds, not just incrementally as new things are added, but logically with a process called tagging.

It’s a popular way to structure the version control system and components so that each component can be tested separately. Here’s the process at a high level:

1. Create release tags for all code and tests
2. Check out, build and test each release tag independently per component
3. Add release tags to external test repositories and perform tests after building
4. Deploy code to the relevant environment when ready
5. Retire tags after enough development has occurred

Let’s discuss each step.

1. Create release tags for all code and tests

In theory, the simplest way to build components separately is for each one to have its own separate place to store changes. This is where source control systems come into play. Subversion, TFS, and GitHub are all examples of applications which provide the tools to manage source code changes. While each version control system has its differences, all of them accept and store changes as they are regularly added. The source control system then tracks the corresponding changes, the lines in which files have new or updated code, as well as when new files are added or removed thereby allowing installation of that code at any time.

Commit is the command used to record those changes to the source control system where it is then often assigned its own unique identifier thereby allowing specific checkout or cloning of the code base at a specific time a commit occurred. The collection created for a group of commits managed by a source control system is called a repository. A repository is a container setup inside a version control system for managing all the details about what has been committed. This enables logical development of each component or app from its own repository to assist with development, testing, and delivery.

All of the subcomponents of a single repository app can then be contained within a single repository. However, as the number of components increases, the sheer burden of managing all those systems within one repository makes that approach unlikely. In addition, the components will likely have shared code libraries or want to interact. Creating separate repositories for components to manage features with specific responsibilities helps clear the build, test, and maintenance headache by allowing them to happen in isolation.

The next step after that is to manage each component through a series of tags. In modern version control systems like Git, tags are essentially local branches. A tag might be, for example, the component and the date for the next planned deploy. Programmers can build and test a component on a tag and version the tests right along with that tag. When the code is deployed to production and other services can rely on that code, the code will also be merged into master.

Versioning the tests along with the components makes it possible to rollback a version or hotfix to production and then run some or all of the tests from that previous version. Without this ability, the hotfix of production from last week’s code would run against today’s tests and fail because the expected behavior would not occur.

Most version control systems have a tag feature. It’s a way to associate a specific version with a name, like the sprint name or a feature name. Modern build systems like Jenkins, TeamCity, and Travis CI allow “build kickoff” to happen for any branch or tag when a programmer adds new code. This allows previous tests to act as a guard against potential regression before that code is merged into master.

2. Check out, build and test each release tag independently per component

Package management tools like Node.js Package Manager (NPM) do a good job of helping load or update to the latest version of a dependency based on semantic versioning rules. In C# this is NuGet; in Ruby it’s a gem. Tests bundled with the code in the same repository provide versioning for free, allowing the last check in to have the latest code for that version and the corresponding testware.

A simple checkout and build for a release on February 28, 2019, for one component could look like this:

$ git clone https://github.com/username/component-repo.git
$ git checkout tags/v2019.02.28
$ npm run build
$ npm run test

For Git, this checks out a clone of the repository, including all of its tags and branches, and then switches out the code for the version tag provided. NPM can then run its build and test in separate steps, as shown above.

3. Add release tags to external test repositories and perform tests after building

Now all you have to do is add these commands to the CI build script for the component or service, and the same steps will be performed each time the tag is updated. That’s pretty simple for tests at the unit, behavioral or component integration levels, or quadrant 1 of the Agile Testing Quadrants.

It turns out to be less true for tests that sit in the other quadrants. These tests often exist externally to the component and may be included inside another component that consumes or integrates with it. They frequently are captured in separate or shared test-only repositories that may run only after the subcomponent is built or after service deployment to a testing server.

These tests tend to cover more parts and at higher levels. Some examples include exploratory tests (which require a lot more thinking to automate than the tests written during traditional test-driven development), end-to-end or scenario tests, and performance, accessibility and security tests. These tests are often run after the application is compiled, and perhaps only after it has been deployed, in the case of performance and end-to-end tests. Tests like these cover systems and subsystems, workflows, and interactions between pages and screens, and include authorization and access levels such as different permissions and roles within a single test.

Thankfully, with the use of release tagging, even these external tests are pinned to the relevant release tag and performed after builds.

To put all that a different way: The software can build a component and run component tests, then put the scaffolding in to build the entire system and run any full system end-to-end tests. This can happen for every build or, if the suite is too large, overnight. Some automation test tools, like Ranorex, can create suites that run more tests more often (for an overnight build) and less for any release (just those tests related to a component).

In Mocha, a JavaScript library for running tests, a build step added to the package.json file might be as simple as:

"mocha --recursive "./tests/*.test.js"

On operating systems loaded with the Bash shell, the command is even simpler and leverages the globstar (**) wildcard character to automatically traverse the directory tree, like this:

“mocha "./tests/**/*.js"

For Ranorex, the compiled tests for a test suite targeting your shopping cart integration can be compiled into a ShoppingCartIntegration.exe file and run with:

“$ ShoppingCartIntegration.exe”

Additional configuration for the tests can then be passed in via command line arguments.

Tagging tests, associating them with a release, and then associating them with a specific component is powerful because you can run just the right tests for any release. For example, instead of a distinct directory under tests, you could include the category type in the test file name as “src/**/*.integration.test.js”, or “src/**/*.contract.test.js” as the criteria for running tests.

4. Deploy code to the relevant environment when ready

Once a commit has been built, run and tested by tools, it is ready to be promoted. Some teams promote the code to test servers for humans to explore. Once the build gets a thumbs-up, the code is tagged again and pushed to a system integration test (SIT) environment.

Once that passes, the code is tagged again and promoted to production. With a component approach, some teams can test a component and promote it to production, perhaps automatically. Examining logs of the test run history can be extremely valuable in narrowing down when a problem was introduced, and even by what change.

5. Retire tags after enough development has occurred

As a bonus, issue trackers can also reference release tags, making it easier to trace bugs to their fixes and tests in source control. Tags can last a long time, but if a component continues to have development over a long while, the name “tag/v2019.02.28” will lose all meaning. Beyond providing the timing of the release date, its value diminishes the more tags are added. That makes cleaning up tagsa good idea, which enables easier development as new features are added.

l retire them once enough newer versions are available and the history on those tags is already deeply consumed in other releases. Just remember to look at all the available tags in both your components, service and test repositories, and your issue tracker before you retire them.

Making Sense of the Complexity

Over time, a large software build becomes slow and expensive to maintain. It becomes brittle. Continuous delivery becomes impossible. Automated end-to-end system tests that were designed to decrease the risk of a bad merge actually slow down feedback and create maintenance work.

Organizing the software in terms of components, managed by tags, can transform this complexity into an opportunity. Teams can build, test and release components on a tag, and they can even share code, working on the same code libraries at the same time but on different tags.

Not using Node.js or Mocha? Don’t worry. Most programming languages and unit test runners provide mechanisms for labeling tests and suites, and the release tag method is available on most modern source control tools, which means you can continue to apply the same technique as your choice of language and components evolves over time.

If your organization has decided to practice continuous testing, what’s the next step? What do you need to know?

For one thing, it costs less to begin than you might think. Continuous testing (CT) accommodates a small start, along with incremental growth. You should plan a useful “live” CT run in the first week of operation — maybe even on the first day.

First, you should understand what practitioners generally mean by CT. The focus is not on testing without cessation, as much as the name suggests that. Instead, CT is generally considered in the context of a software development lifecycle (SDLC) where developers commit source code at various times.

The usual interpretation of CT is that each event defined by submission of a changeset triggers a testing cycle. If those tests fail, then the changeset is rejected, and the application reverts to its last accepted state. If we’re being literal, the practice may better be called low-latency, event-driven, or fast-feedback testing, rather than continuous testing.

One variation on this theme calls for testing to occur on a definite schedule, except only as needed. An organization might decide, for instance, that the repository be scanned every two hours during the work week; if an update appears, then a test cycle begins.

An even more important refinement of the CT intent is to define different scales of testing. For instance, an organization might schedule these three scales:

• Every minute, a collection of quick regression unit tests fires
• Every time a commit is made, certain relatively fast integration tests are launched, and the commit is reverted if any exception turns up
• Every day at half-past midnight, a collection of time-consuming performance, security and integration tests runs

We design combinations like this to make the most of the expenses of testing. The payoff, of course, is to identify an error as close as possible to the introduction of that error.

An incremental approach

Clarity about these different possibilities encourages an incremental approach to CT. What’s your single most common or damaging category of error that slips through your organization’s existing workflow? That’s an ideal place to start.

If there have been occasional transient errors in authentication that show up inconsistently for different users, you can design a big exercise to run an appropriate minute-long, end-to-end exercise for each of a hundred users. Automate and launch that test every morning at 1 a.m. and results will be available well before 3 a.m. That becomes your first step in CT, and the team can build from there.

Or is it too common for programmers to check in code that looks right, and even passes peer review, but it ends up working only on specially configured hosts? Define the right unit test, iterate it over the full range of pertinent platforms, and block acceptance of any changeset until the regression unit test suite completes. All of them together shouldn’t take more than a few seconds.

A first step need not be perfect or comprehensive; it’s more important to be meaningful, well understood, and easy to maintain. A good CT plan starts small, with early wins, and promotes extension and expansion.

It’s also essential that CT have “teeth” from its first run because if CT results can be ignored, they will be. A failure of a CT test needs to have a real consequence — generally to block acceptance of a deliverable. CT can’t be on the side of your SDLC; it must be in-line.

CT isn’t a product or commodity, and there’s no way (for now) just to order CT as a line item. For at least the next few years, someone with a specialized focus will need to tie together a repository, testing tools, test data, and probably a notification or ticketing system in order to implement a useful CT installation.

Opportunities for CT

Two special circumstances deserve consideration in any CT plan.

First, look for places throughout the SDLC to apply CT. While it’s most common to apply CT in the vicinity of code check-in, other SDLC points are possible. A product with a complex generation procedure might define CT with unit tests on source commit, then a later stage of CT to validate generated products.

It’s also possible, at least in principle, to have CT without full automation. As one aspect of incremental management, it’s sometimes wise to define a CT “gate” before the gate is fully mechanized. A plan might call for verification of a human-readable result of a complex operation, for instance. It’s OK to stub one CT step so that a trusted testing specialist views a result and pushes a button to accept or reject it. While such a step should be automated as soon as possible, it can be valuable to make the test part of a larger routine from early on, even before full automation.

To summarize, plan your CT approach ahead of time, starting small with easy wins; but make sure you can expand your plan later. Be sensitive to different scales of CT, particularly across time ranges and platforms. Look for the errors that bring the greatest business risk and figure out how to create CT that manages that risk. And recognize that CT itself likely will require daily maintenance and refinement, but that daily effort pays off in a stronger overall test strategy.

Machine learning (ML) is expanding explosively. ML’s applications are so diverse, though, that testing that makes perfect sense in one situation might be pointless in another.

If you work in the internet of things, legal technology, the financial industry, or several other regulated fields, you need to be careful to distinguish the scopes of software testing and scientific testing for your models — and to do both well.

Machine Learning Oracles

Consider the credit-card industry. Processors invest a great deal in fraud detection and mitigation. Machine learning is one of their techniques: in broad terms, automated detectors are trained to conclude from certain combinations of data that particular charges are likely to be fraudulent. ML appears to be indispensable in this domain because of the volume and complexity of data involved. The above linked article concludes with a 52-arrow flow chart visualizing an ML algorithm; it’s characteristic of expectations in this area that the article’s author says the model is explained “very simply.”

In this technique, the training is on a control set. The ML is then, ideally, validated on a different data set or test set. This is a situation that matches conventional software-testing approaches: Given a fixed control set as an input, does our algorithm provide consistent predictions about the test set? Can we use the data as a regression test to validate the invariance of certain results?

The deliverable of this kind of ML is an oracle that learns from an entire population of characteristics pertinent to all members. ML in other domains acts to distinguish photographs of cute kittens from terrorists, or to diagnose illnesses based on medical histories, or to predict when transport engines are best overhauled. When new data arrives, the ML improves the “knowledge” of a single central oracle that applies to the entire population.

A second, less-publicized kind of ML acts on individuals. A product in this category might receive data from a particular bank on its check transactions, apply ML, and predict fraud or other problems about checks received by that single bank. There’s no assumption that the oracle for one bank works for another.

Multiple Oracles in Play

This turns out to be a harder testing problem. The product in this case is actually a generalization of the first kind of ML: The software doesn’t just create a single oracle from a control set, but it must be capable of creating a different oracle for each of a universe of control sets. This additional level of abstraction makes testing combinatorially harder.

More than that, testing is more subtle. In the first category, where the outputs are fraud predictions, outputs have a simple score based on the number of frauds correctly detected and missed in the whole population. The quality of the second product has to aggregate the qualities of the oracles produced by the ML for each individual bank.

Another way to express this difficulty is to note the decoupling of the software testing from the scientific testing. Suppose the oracle generated for a particular customer gives bad predictions. It’s possible the software perfectly embodies the required ML, but the ML applies so poorly to this customer that it produces a low-value oracle. It’s equally possible that the ML is sound, but the implementation mishandles an edge case that previous quality assurance didn’t detect.

While regression testing the second category demands richer data sets, just to multiply instances is unlikely to be sufficient. Statistics isn’t linear, in the sense that inferences don’t scale on a straight line: To double the quality of a test might, depending on the details involved, take four times or 60 times as many data points, or even more. Separate oracles for separate consumers is an abstraction whose testing can only be tamed through introduction of more introspection on the oracles.

Introspection, in this case, means that a trained oracle not only gives an answer — “This check is unusually likely to be returned for insufficient funds” or “Replace the diaphragm assembly now” — but also insight into the facts that led to this conclusion. While the oracle might not make available to the end-user intermediate propositions such as “The check was presented to a small bank out of state” or “Phosphorus level were more than one standard deviation above average,” it’s crucial for quality assurance purposes that those intermediate propositions be tested, along with the final results. That’s generally the only effective way to control the extra level of ML abstraction.

Securing Adequate Testing

Conventional ML can be tested much like other software. While the data involved might be particularly large or complex, in principle it’s still possible to synthesize appropriate regression tests for this kind of ML.

ML that produces a customized oracle for each client or end-user, though, demands more specialized and sophisticated testing. Usual coverage and quality scores can be high, yet many functionally significant computations may be left unchecked. A deeper background in data science is usually necessary for adequate testing of this second class of ML.

If you pay the costs of test-driven development and agile practices, then Extreme Programming and related schools promise several benefits, including quicker time to market and lower technical debt. However, practitioners often argue about the benefits of TDD. Academic research mostly has concluded that those benefits are ambiguous.

While TDD is often treated as a monolith whose adoption constitutes a boolean choice, there are two specific and closely related challenges when it comes to the realization of TDD: mocking and dependency injection. Clarity around these aspects, including their costs and techniques for addressing them, gives development teams at least an opportunity to plan more accurately — and perhaps even practice a more rewarding variant of TDD.

Automation and integration

First, a few definitions. TDD is the practice of software coders writing executable tests before implementing requirements. A number of concepts, terms of art, tools, canons and other cultural items, including the “red, green, refactor” slogan, commonly accompany TDD.

As TDD enters at least its third decade, it also has controversies that have themselves become conventional: how to apply TDD to legacy work, whether TDD makes sense for GUIs, what TDD means for mobile or real-time apps, and so on. This article leaves all those subjects aside, as others have already covered them adequately. The focus here is entirely on the contrast between two “flavors” of computing: automation and integration.

Automation encompasses algorithms, like Euclid’s algorithm, the wavelet transform, and the traveling salesman problem. Automation generally looks more or less like scholastic mathematics, and TDD works great for automation. TDD is an appropriate way to express and validate edge cases and essentials and variations of automation. Algorithms and related automations constitute typical curricula of formal computing education, and they dominate the industry employment interviews of folklore.

Integration is intellectually far shallower. It’s mostly just recognition that information here is needed there, and flanging together a few parts to effect that move. TDD sometimes gives us only clumsy help with integration, and it often appears to be more trouble than it’s worth. That’s unfortunate, because a growing fraction of programming work looks like integration.

An example helps. Imagine a requirement for a particular function to retrieve a sales index from a database. There’s nothing deep about this; the database lookup is intended to be as simple as possible. An implementation might look like this:

def get_sales_index():
handle = connect(DATABASE, USER, PASSWORD)
cursor = handle.cursor()
cursor.execute("SELECT special_value FROM special_table")
datum = cursor.fetchone()[0]
handle.close()
return datum

While this is Python, the corresponding implementations in Java, Ruby or JavaScript look quite similar. This is a good implementation in several ways: it’s brief, it doesn’t leak memory, it’s correct (at least to the extent that exception-handling hasn’t been specified), and it’s readable to those familiar with database culture.

It’s bad for TDD, though. TDD teaches that unit tests are small and self-contained. To test get_sales_index(), though, requires connection to DATABASE. DATABASE is outside this source. Potentially worse, DATABASE might be a live production database, to which tests or testers aren’t allowed to connect.

Solution?

Things rapidly become more complicated at this point. The usual treatments of TDD introduce mocks and dependency injection, and our original get_sales_index() becomes something like

def get_sales_index():
return get_sales_index_kernel(get_cursor(DATABASE, USER, PASSWORD))

def get_cursor(DATABASE, USER, PASSWORD):
return connect(DATABASE, USER, PASSWORD).cursor()

def get_sales_index_kernel(cursor):
cursor.execute("SELECT special_value FROM special_table")
datum = cursor.fetchone()[0]
handle.close()
return datum

Each of these three definitions admits at least the possibility of a realistic test, in principle. Notice the tests themselves, which this article omits in the interest of brevity, have become rather arcane, with dependence on mocks.

This example is entirely typical for what happens when testing integrations. Single entry points multiply into two to four linked methods, with associated mocks and dependency handles also proliferating in the associated test code. It’s easy to conclude that the cure is worse than the disease.

But we need some solution! More and more of day-to-day programming is integration, and it needs to be reliable.

Clarify processes

There is no single solution. The best we have, at least for now, is more clarity about how to make the most of TDD. Keep these tips in mind:

• Expect to test all your automations without mocks.
• Recognize that construction of mocks to test integrations can require more effort than writing the integrations themselves.
• Consider configuration of testing resources — a testing database, testing message queue, testing network, and so on — as mocks. It’s increasingly cheaper to bring up another database instance than to try to specify enough database behavior to define a useful database mock.
• Share mocks among tests. Define as few of them as necessary, but have those few be robust.
• At an architectural level, consider relaxing TDD’s usual tenets. Work out a good dependency injection style for the project, make sure all the coders understand it, and establish an explicit target that combines unit tests, acceptance tests, static code analysis and peer review to achieve high quality in the integration parts of the project.
• The real cost and benefit of mocking aren’t in first implementing functionality, but in maintaining it. If your mocks help maintain invariants during refactoring, you probably have good mocks. If, however, you find that you have to update the mocks each time you write new functionality in order to keep regression tests green and passing, then your mocks might be more trouble than they’re worth. It’s probably time to adjust your testing strategy in that specific area of integration.

There are several reasons you may decide to begin continuous testing: earlier opportunities for QA, increased risk mitigation, and the ability to keep up with faster dev sprints. And if GitHub also hosts projects you’re working on, you’re in luck! Plenty of others have passed this way before, and you’ll find the path forward is relatively smooth and well-lit. But a few preliminaries will help your voyage.

Tooling up

First, much of the existing literature refers to CI (continuous integration) or even CD (continuous delivery or continuous deployment) rather than CT, or continuous testing. Even if you know your focus is specifically on CT rather than the broader span of business aspects that CI and CD cover, be prepared to read up on CI to get the information you’re after.

Don’t be distracted by how often release candidates are generated, or disputes about elaborate rules for administration of deployments. Your attention is just on the CT part: You want to make sure each commit launches its own regression test.

A second cultural point to make is that while tooling is important, CT tools are rather convergent in their basic capabilities. When starting out with CT, it’s generally more valuable to make a reasonable choice swiftly than to try to decode the abstract technical capabilities of different competitors in relation to your project’s specific requirements. Anything you can do to comply with existing organizational standards or to leverage the experience of your team is more likely to pay off than an exhaustive search for technological compatibility.

If your organization already has licensed CI from a particular vendor, use that. If you happen to belong to a great user group where you can learn from experts with one specific tool, use that. In any case, choose a CT tool, make the most of it, and begin to enjoy the benefits of CT. If, some day in the indefinite future, you find that your specific choice has a limit you can’t work around, much of your CT scripting or configuration will still apply within alternative frameworks.

A different way to say the same thing: Most of the better CT tools allow for integration with GitHub and common CI processes. Expect that you’ll advance rapidly once you start your CT work.

CircleCI example

CircleCI is one of several frameworks that make initial CT experiments easy. If you have an existing GitHub project, CircleCI lets you sign up with those credentials.

It’s crucial in your first quick CT configuration to start small. CircleCI and competing services are mind-bogglingly flexible. Don’t try to understand everything they can achieve; focus instead on the simplest possible configuration that applies to you.

Refer to this demonstration site to confirm your steps.

The recipe

Add a new file .circleci/config.yml with content
version: 2.1

jobs:
build:
docker:
- image: circleci/php:cli

steps:
- checkout
- run: echo '<?php print "CT success."; ?>' | php

to your repository. Are you not a PHP programmer? That’s OK; once you finish this first exercise, it’ll be quick work to change to a better fit. The aim for now, though, is to work with something as concrete as possible, so you can start to experience CT for yourself.

With config.yml in place, visit CircleCI, and log in using GitHub — you’ll see a button labeled exactly “Log In with GitHub.” That takes you to your dashboard, where you’ll need to follow your project. As soon as you follow your project, your dashboard should show your project next to a green “SUCCESS” label.

That’s it! As the label puts it, you’re a SUCCESS.

What to do next

While that first step is the most important, it is only the beginning. Next you’ll want to view the log of your SUCCESS, which should finish with the result

...
CT Success.

Experience the value of CT by introducing an error into the configuration; you might, for instance, change the run line to

- run: echo '<?php printx "CT success."; ?>' | php

Soon after you commit this to your GitHub repository, your CircleCI dashboard will light up red with the complaint “… syntax error …”

Do you see the value? As you add real tests to your configuration, they’ll automatically run against every commit. Never again should it be a surprise for an error to sneak into a branch, because every commit will be tested.

Next up

You’ll make rapid progress from this point:

• Swap out PHP for a language environment that fits your own needs better, or even customize a Docker image for your project
• Introduce tests you’ve already written into your configuration
• Define rules for specialized testing scenarios or reports on who gets what test results
• Read up on jobs, workflows and orbs so you make the most of the freely available artifacts other teams share publicly

Keep in mind that essentially everything in your configuration is variable, and even variably variable. Start small, get something working, and keep improving it in the direction of your vision of what CT should be for your team.

Minimize your configuration — that is, while your CT configuration language has enough power to do almost anything, you’ll generally be better served to script tests and filters outside the CT, and just use the CT to “glue” together working parts. With a little practice, powerful testing tools integrate nicely in nearly all CT schemes.

Right now, wherever you’re starting, you’re just a few minutes away from dramatic CT progress.

It’s important that tests be automatic and thorough. Continuous testing is an approach for automated testing that gives fast feedback by testing early and often. But “continuous” doesn’t mean all tests need to run at all times. Here are four tips for organizing your continuous testing system.

1. Know yourself

Continuous testing (CT) is valuable to essentially all developers. But different developers are different enough that the value will vary a great deal. Run CT for the specific benefits it brings in your situation, not just because CT is a hot topic right now.

Programmers compiling Java for million-line embedded automotive modules have a different focus from freelance mobile app developers, and both their needs are distant from those of a data scientist writing Python to turn around proteomic results quickly. What are the metrics of your own situation?

How often do your programmers commit updates? How long do your unit tests take to run? How long is your build process? How long do your automated integration tests take to run? How often can your quality assurance (QA) crew test a release? How many committers are active?

Before you launch a CT initiative, measure all these aspects of your process, at least crudely. Your plans will be better for being based on reality, rather than your impression of what you and your team do.

2. Everybody runs unit tests

Unit tests need to be automatic, cheap, and easy. If they take 40 minutes to run, they’re not unit tests. If they can only be run on certain hardware, they’re probably not unit tests. Tests that demand human oversight—such as “Is that click in the right place on the screen?”—are not unit tests. If a supervisor has to enter a secret passphrase at a front-line contributor’s desk, the context is definitely not for unit tests.

That doesn’t mean those other tests aren’t valuable; they might even become unit tests some day. One of the first actions to implement a useful CT plan, though, is to identify a kernel of all your tests that are automatic, cheap, easy, and ready right now. Segregate those as unit tests.

Make sure any commit that fails a unit test tosses an alarm, and make sure all your programmers know how to run the unit tests for themselves. With those two elements in place, your CT will begin to pay off, even if the unit tests cover only a small fraction of all the tests you eventually configure.

3. Select which tests work for continuous testing

Next up, identify tests that are feasible for your developers but not your CT system, or vice versa.

How can that be? Here are examples:

• An important test takes eleven hours to run. While that’s impractical for humans, and shouldn’t be allowed to stall results from your CT’s examination of individual commits, your CT can execute such long-running tests in the background and report results at least a couple of times daily
• For a variety of reasons, your CT might be on a network that is distant from developers and lacks certain resources. Maybe security restrictions mean that the CT can’t access testing databases, at least for several months, but developers and QA can run through a whole suite of database-dependent tests in 30 seconds. In such a case, establish a policy that humans are responsible for running these integration tests. Create a plan for a proxy, new resource or something that will eventually make the test possible for CT
• Certain tests might appear to be available in any environment, but licensing restrictions effectively confine them to a mechanical schedule. Run such tests using CT. While not being able to run them directly might mildly frustrate programmers, they’ll appreciate the deterministic results from CT and find ways to fix errors that turn up

Not all tests deserve to be in the same category. Manage tests with enough care that the whole team understands which are unit tests, which are integration tests only run by CT, which are integration tests for any environment, and so on.

4. Plan for the future you want

Does your testing plan still have gaps? Are there error-handling segments of source that are hard to test, or tests that require human intervention and judgment, or errors that show up only for end-users?

Like anything in life worth doing, your CT’s unlikely to be perfect at the beginning. Document your gaps, and figure out which ones will reward correction the quickest and best. Go slow on tests that promise little return, and learn to recognize tests that cost more than they’re worth.

Whatever you do, uphold the integrity of your CT: Software can’t be released unless it passes tests. Don’t let testing become a spectator sport or advisory pursuit; cultivate a culture that understands tests must “go green” before software merges or releases. Do these things, and CT will pay off for you.

Does “testing in production” (TiP) sound like crazy talk? Does the thought of working right next to customer data sound as comfortable to you as cooking dinner next to a crocodile pool?

Those are healthy instincts. But testing in this manner doesn’t need to be a risk-filled activity. Here’s what you need to know so that TiP becomes not just possible, but sensible.

I earlier wrote about reasons to test in a production environment and how to make the most of it. As I mentioned there, though, TiP is a rich subject and deserves deep study.

Still, there are definite ways to set yourself up for success. (Note that the points here apply best to TiP for software as a service (SaaS); while installed and embedded software have their own related TiP concepts, they’re different enough to treat separately.)

Take these five basic steps now to make sure you’re well equipped for your TiP journey.

1. Know your load cycle

The ideal for TiP is to do it so carefully and perfectly that it’s safe. Computing systems always change, though; if only for that reason, wise engineers plan on imperfection.

One way to do that involves knowledge of your production systems’ habits through time. Does your load go down on the weekend? Does it go up then? How do your customers’ needs vary over a 24-hour day? Do holidays or weather events consistently shift demand for your service?

Awareness of customer habits helps you find favorable times on the calendar for testing. You’ll want to run many tests when other demand is light. Certain kinds of user tests, though, go best when spread over as many distinct users as possible, for as short a time as possible.

You’ll be in the best shape to manage these schedules when you have clear and accurate details about what your customers do.

2. See what your customers see

Provide testers and support staff with whatever is needed to see faithful representations of what customers see. To make workers guess or ask clients how things look on their side just introduces noise, confusion, and delay.

Make sure they can access the browsers, hardware or remote-control facilities, or other resources needed to share views meaningfully. If privacy is a constraint, build in a way to experience what clients experience, except with crucial details blurred or anonymized. Looking at the same picture is crucial for understanding customers’ experience.

3. Get security help

Security considerations change rapidly. Your service needs a security specialist both to review the security consequences of your TiP plans and to advise about security tests to integrate into your TiP program. Find a way to adopt DevSecOps, or at least bring in someone who can be trusted to ensure your TiP doesn’t harm your security profile.

4. Use multiple test accounts to model different customer profiles

Left to themselves, many developers and even testers default to the use of personal accounts with elevated privileges for testing. That’s not good enough. Testing at least part of the time with customer-level accounts is essential.

Don’t leave this to chance. Explicitly construct test accounts that model the personas your marketing department has defined or your support staff has profiled. While use of a personal account that’s reconfigured has its place, don’t rely on that for your formal test plan; assume that testers need to manage a variety of different accounts with well-defined properties.

If your system doesn’t support “extra” test accounts, that’s a problem that must be solved. Exercising a system with a variety of profiles and permissions is fundamental to testability, however many organizations neglect it.

5. Learn TiP’s fundamental techniques

To teach all that’s necessary for TiP is beyond the scope of this introduction. The point for now is that TiP is distinct from more conventional testing. Your study of TiP will make much more sense once you clearly understand some fundamental techniques:

• A/B testing
• Canarying
• Distributed tracing
• Feature flagging
• User engagement testing

Learn these techniques in whatever way makes sense in your situation: sit in a college classroom, read some books, write your own implementation, attend a conference, organize a talk series for your team, or whatever is effective for you. To progress with TiP, you need to take advantage of those who know more, and it’s easiest to understand them when you share a basic vocabulary.

Make TiP a strength

If TiP seems risky or intimidating, remember: You’re already testing in production. With a little care, you can do it well: manage your plans and results, enhance security and stability, and make TiP a strength rather than an accident.

Recognize that testing is always a team sport. Even this brief introduction to TiP gives ways to align with colleagues in marketing, support, development and infrastructure. You go farther and better when you learn from others, and especially when you learn the values they see in TiP.

This article describes how to use the Ranorex Studio IDE and the Ranorex API for test automation in your behavior-driven development (BDD) process. BDD requires a cognitive shift: instead of thinking about testing functions, you are now thinking about behaviors. Following this approach enables seamless collaboration between business stakeholders (such as product management) and the QA or development teams. On the business side, the product management team can define feature specifications in natural language without writing a single line of code. On the technology side, the QA team and the development team add automation steps using recording modules and the repository.

Tool Stack

The diagram below shows a sample tool stack to create a BDD-based testing framework. Ranorex Studio provides the development environment, while the Ranorex test automation framework provides the automation API. To parse the feature files from natural language to written code, we are using SpecFlow as the BDD interpreter. Finally, NUnit is the unit test provider, which manages and executes the tests.

Process Overview

When following the BDD approach in an agile environment, the entire development process of a user story is divided into a business side and a technology side.

On the business side, we create user stories, which are taken and broken down into features stored in feature files. These features are broken down into scenarios. The features are written down in a natural language following the Gherkin syntax. A basic feature written in Gherkin might look something like this:

Feature: Serve coffee
In order to earn money
Customers should be able to
buy coffee at all times

Scenario: Buy last coffee
Given there are 1 coffees left in the machine
And I have deposited 1 dollar
When I press the coffee button
Then I should be served a coffee

On the technology side, we need an interpreter to parse the feature files written in natural language (Gherkin syntax) and call test automation code. These Ranorex API calls or Ranorex recordings do the following:

• Automate the system under test
• Verify the expected output
• Returning a corresponding result

Prepare Ranorex Studio for BDD

The SpecFlow Add-in for Ranorex Studio

The SpecFlow add-in provides file templates for feature files, step definition files and event definition files. It also translates the features from Gherkin syntax to C# code. To install the SpecFlow add-in to Ranorex Studio, follow the instructions below:
Download or clone the SpecFlow repo and open the project “TechTalk.SpecFlow.RanorexIntegration.csproj”

You then need to ensure that the project is targeting the .NETFramework version 3.5. You will notice that some NuGet packages are missing, as indicated by the yellow warning symbol. Use the context menu and select Manage NuGet Packages, then click restore. Add the NuGet package for NUnit.Runners.2.7.1, it is important that you use version 2.7.1

The packages will be restored from your online sources, until you are left with only three warning symbols. The following packages will not have been restored and need to be removed, in this example using the context menu.

Then replace them using the .dll files found in <Ranorex installation folder>. Click on references and use the context menu to select add reference. You may need to click browse and navigate to the Ranorex installation folder.

Next, build the RanorexIntegration add-in.

A .sdaddin file will then be built in the IdeIntegrations folder. Use an extraction tool such as 7-Zip to extract the file and save it to a new folder named “TechTalk.SpecFlow.RanorexIntegration”.

Copy the extracted folder and paste in the <Ranorex Installation folder>Bin\Addins\Misc.

Make your Ranorex Solution BDD ready

In your Ranorex solution open the “Package management console” (View->Tools->Package management console) and run the following commands:

Install-Package SpecFlow.NUnit -Version 2.3.2

Install-Package NUnit.Runners.2.7.1

You will notice that the references are added to the projects view. To ensure that your BDD solution integrates seamlessly with SpecFlow and NUnit you will need to amend the AssemblyInfo.cs and app.config. Simply double click on each in the projects view.
For the AssemblyInfo.cs add:

[assembly: NUnit.Framework.Apartment(System.Threading.ApartmentState.STA)] 

as seen below:

For the app.config you will need to add <probing privatePath=”Runtime”/>
in ‘assemblyBinding’ as seen in line 17 of the below.

Get started with BDD

Now that your Ranorex Solution is BDD-ready, you can begin defining features in SpecFlow and creating automated tests for them in Ranorex.

Define a feature from the business side

In SpecFlow, create a new feature file. To do so, right-click on the project, then choose Add > New Item > Category: SpecFlow > SpecFlow Feature.

Create step and event definitions for the technology side

Create a new step definition file. To do so, right click on the project and choose Add > New Item > Category: SpecFlow > SpecFlow Step Definition.

Now, create methods for all “Given,” “When,” and “Then” actions described in the scenario description. Add “Given”, “When” and “Then” attributes with values matching the scenario steps defined in the feature file. To do so, follow the samples provided by the template from which the new step definition file has been generated.

Save the file. Ranorex Studio will create a <name>.feature.cs file containing the automatically-generated code that will execute the test scenarios using the configured test provider.

If the code/C# file is not created please refer to the troubleshooting section at the end of this blog. 

Create step and event definitions for the technology side

Create a new step definition file. To do so, right click on the project and choose Add > New Item > Category: SpecFlow > SpecFlow Step Definition.

Now, create methods for all “Given,” “When,” and “Then” actions described in the scenario description. Add “Given”, “When” and “Then” attributes with values matching the scenario steps defined in the feature file. To do so, follow the samples provided by the template from which the new step definition file has been generated.

SpecFlow searches for the step definition text in the step definition files that matches the text in the scenario step.

Now you can implement the automated step, either by creating recording modules or by manually implementing the steps using the Ranorex automation API.

Your implementation might look something like this:

You can create additional steps to be executed before and after a scenario, such as performing setup and tear down tasks, including starting and closing the system under test.
These steps can be defined in an event definition file and might look something like this:

If your assembly cannot be found please see the troubleshooting section at the end of this article for further details. 

Run a BDD test using NUnit

Because this solution uses NUnit as the unit test provider, we use a NUnit Runner to execute our tests.
Your command line call should look something like this: (Use root folder)

..\..\..\ .\packages\NUnit.Runners.2.7.1\tools\nunit-console-x86.exe <solution>.dll

Conclusion

The preparation of Ranorex for BDD is complete. From this point forward, your business team can define user stories in a natural language following the Gherkin syntax, and your development and QA team can implement the automation steps in the step definition files.

Troubleshooting

The code/csharp file is not generated

If the code file is not created in Ranorex Studio, you have to set the “Custom tool” in the files properties to “SpecFlowFileGenerator” to make Ranorex Studio create the code file.

Assembly not found

If your assembly was not found you will need to build your project as a library. To do so, open the project properties and set the output type of the project to “class library”. In addition, copy all Ranorex runtime DLLs to the output folder.

Test cases are an essential part of any software testing process. They help us determine if features of an application or system under test work correctly. The efficiency of testing depends on how well we manage our test cases.

A test case can have the following properties:

• Test case ID — Usually a numerical value to identify a unique test case
• Test case summary — The summary/description of the test case, for example, “Check customer login with valid data”
• Preconditions — Any prerequisites that must be fulfilled before the execution of the test case
• Test data — Variables and their values that are to be used while conducting the test, for example, a user ID and password
• Test steps — Step-by-step procedure to execute the test, e.g. open a website, enter user ID and password, click the ‘Submit’ button, etc.
• Expected result — The expected result of the test, e.g. User is successfully logged into an application
• Actual result — The actual result of the test, e.g. login was successful/unsuccessful
• Pass/Fail — Marks whether the test case passed or failed
• Comments — Any comments on the test case or test execution.

Of course, additional properties may be included depending on the policy or method used by the particular team to create test cases.

Different policies may also determine how teams should document, execute and track test cases, in a process known as test case management.

Test case management is essential for the efficient testing process because it helps testers, developers, managers, and other team members to keep track of test cases, as well as test strategies, test plans, and test runs in detail.
Failure to document test cases may result in test steps being unnecessarily repeated or, worse, missed completely.
There are a number of ways to manage test cases, and some are more efficient than others. For example, we can use spreadsheets to add test case properties manually or track everything we test in a bug tracking tool, such as Jira, using a custom issue type.

But these two methods include some problems. Using Excel or Google Docs to manage your test cases may work on a small scale. However, as your team or app grows, this approach can become troublesome. The spreadsheet document may get too long and become hard to filter/search. Also, there would be no execution history (pass, pass, fail) and you may find reporting too cumbersome.

The problem with issue tracking tools such as JIRA is that they can be configured to support test cases, but they do not provide testing-specific features, such as reports, execution history, etc.
Furthermore, using either of these methods means that we need to create and track our test cases manually, which will definitively slow us down

This is why many companies resort to specialized test case management tools such as TestRail to track, maintain, and analyze their test cases.

About TestRail

TestRail is a web-based tool that helps developers and QA teams to easily manage and track their manual and automated test cases and test runs.

The tool covers important aspects of software testing, ranging from simple tracking of each individual test status to detailed planning and coordination of the entire testing process.

Thanks to informative dashboards, metrics, activity reports and email notifications, TestRail provides real-time insights into testing progress.

One of the greatest strengths of TestRail is the ability to integrate with many popular project management and issue tracking tools, such as Atlassian Jira, Bugzilla, TFS, GitHub, Visual Studio, FogBugz, Assembla, as well as test automation tools, including Ranorex Webtestit.

You can check the official documentation to see how each of the supported tools integrates with TestRail.

Integration with Ranorex Webtestit

In this article, we’ll use TestRail integration with Ranorex Webtestit to explain how to easily manage your test cases.
This integration will allow us to import TestRail projects into Ranorex Webtestit and then export test reports back to TestRail once the test execution in Ranorex Webtestit is completed.

Enabling TestRail API is the prerequisite for this integration, so the first step will be to open TestRail, then go to ‘Administration’ > ‘Site Settings’ > ‘API’ and check ‘Enable API. ‘ We’ll click the ‘Save Settings’ button to complete the action.

Also, as we are going to import some TestRail projects into Ranorex Webtestit, we should first create them (if we have not created any).

In the ‘Administration’ section, select ‘Projects’ and then click the ‘+ Add Project’ button.

On the ‘Add Project’ page, we can add project details, such as the project’s name, user roles that can access the project, defects, references, etc.

After we’ve created a project in TestRail, we’ll be able to manage it from the TestRail Dashboard.

Adding test cases to a project will be the first step. To do that, go to the ‘Test Cases’ tab and click the ‘+ Add Test Case’ button.

Of course, we can create multiple test cases in TestRail projects. For each test case, we can add details, such as title, type, priority, preconditions, test steps, etc.

After we’ve added all test cases to our project, we can prepare things for a test run by clicking the ‘+ Add Test Run’ button.

TestRail allows us to add additional details for our test run such as name, milestone, to whom it will be assigned, etc. Also, here we can select test cases that will be included in a test run, either all of them or specific cases.

After setting all the required prerequisites in TestRail, we can open a new project in Ranorex Webtestit and enable integration with TestRail.

To do so, we will open preferences dialog by clicking on ‘Ranorex Webtestit’ > ‘Preferences’ > ‘TestRail integration.’

In the Preferences dialog, we will enable the integration and then enter our TestRail’s profile URL available in the following form: https://yourHostURL.testrail.io

Of course, you need to replace ‘yourHostURL’ with the web address of your TestRail account.
After clicking the ‘Connect’ button, we will be prompted to enter our TestRail credentials, e.g. our email and password. If the URL and credentials are valid, the green ‘Connected’ status will appear in the dialog.

We can also specify the ‘Project ID’ in the Preference dialog, but we can leave this field blank and choose the project later while importing TestRail projects.

Click the ‘Save and Close’ button to close the Preference dialog and complete this step.

After saving the changes, the ‘Import from TestRail’ icon should be visible above the project tree. This icon is only available if the TestRail account is connected with Ranorex Webtestit.

Clicking on the icon will open the import dialog where we can select the project to be imported from a dropdown list. The tests are organized in the project’s suites, sections, and sub-sections. We can also use the filter to only show types of test cases that are automated.

The imported test file will contain information about the test case and test suite IDs, as well as a direct link for opening the test case in a browser.

If you’ve added test steps while creating a test case in TestRail, they will be formatted and displayed as comments in the test file.

We’ll then create the test steps in the imported test file, and then add an endpoint to start the test execution.

Once the test execution is complete, we’ll have an option to export the results back to TestRail. To do that, simply open the test report and click the ‘Export to TestRail’ icon.

You can customize the export format for the test run in a new dialog. Choose the format you want, hit the ‘Upload’ button and the selected test result will be uploaded to your TestRail account.

We can jump to the test run using the ‘View in TestRail dashboard’ link in the export dialog…

… or directly from the report.

The imported tests will be available in the ‘Test Runs & Results’ tab on the TestRail dashboard.

Clicking on each test will display the report with all the important details about a particular test run.

TestRail and Ranorex Webtestit can also integrate with Jira, which allows us to directly create bug tickets if any of the test runs failed or produced errors.

TestRail also lets us customize pretty much everything in it, including test case fields (properties), case types, templates, priorities, etc.

QA teams may find TestRail ideal for their workflow as it supports project collaboration between multiple users with different roles, with an option to add more custom roles and permissions.

Conclusion

The efficiency of testing depends on how quickly and easily we can create test cases, run tests, and track results. In this article, we showed how easily you can manage multiple test cases and test runs if you integrate TestRail with your test automation tool.

Here, we explained the integration with Ranorex Webtestit, which also provides a lot of features and integrations essential for seamless testing.

Maintaining tests, keeping test cases current, and finding and fixing issues is very time-consuming. If you’re using Atlassian Jira for issue tracking, you can ease the burden of test maintenance by integrating Jira with Ranorex Studio for test automation.

Ranorex engineer Jason Branham recently demonstrated the integration in a live webinar, which we’ve summarized here for you. If you prefer, you can watch the entire webinar (including the extensive Q&A portion) on-demand. The webinar timings are shown below so that you can jump right to the topic that most interests you.

• Introduction to Ranorex Studio: 1:00
• Configuring the integration: 4:06
• Opening a new Jira issue for a failed test case: 14:26
• Updating a Jira issue when the linked test case succeeds: 23:30
• Live Q&A: 28:24

Introduction to Ranorex Studio

Ranorex Studio is an all-in-one tool for desktop, web, and mobile test automation, with a user interface for codeless automation of your test cases as well as a full IDE. If you’re completely new to Ranorex Studio, you can learn more about it from one of our introductory videos on desktop, web, or mobile testing.

In Ranorex Studio, the major unit of organization is a test suite, which can contain one or more test cases. In turn, a test case is made up of reusable modules that basically function as a keyword-driven test automation framework. For example, a test case to add a record to a database might include a module to launch the application, log on, insert the record, validate the record, and then log off. Using the codeless approach in Ranorex, you can easily create a new test case by combining these reusable modules, as shown in the image below.

Configuring the integration with Jira

Ranorex Studio provides a wizard for the integration. To access it, open a test solution, go to the main menu bar and choose Tools > Jira Integration > Start Jira wizard.

A dialog box appears, where you can enter the URL of your Jira instance. For purposes of the integration, there is no difference bewtween on-premise and cloud instances: they work the same.

Enter a valid email address and API token for a user account for your Jira instance. Ranorex recommends that you create a user specifically for the Ranorex integration, so you have a dedicated email address to receive test notifications.

After entering your credentials, Ranorex Studio saves them for use in future test solutions. You won’t need to enter them again, even if you use a different browser for testing. To change your Jira connection in the future, simply use the Switch account option in the wizard.

After connecting to your Jira cloud or server instance, you’ll see existing Jira projects based on the privileges of your Jira user account. Choose the project that you want to integrate with Ranorex Studio. You’ll also be prompted to pick the issue type that you want Ranorex Studio to use when creating issues. This is all that’s needed to enable manual issue handling.

The next step is to enable automatic issue handling. The values in the drop-down fields are based on your Jira project. You can specify the state of the new Jira issue when a test fails, such as “To Do,” and the state the Jira issue should have if the test succeeds in the future, such as “Done.” 

After the configuration is complete, you’ll see a new file named Jira.jsd in your Ranorex solution. If your Jira instance requires additional fields, you’ll be promted to enter these in your Jira.jsd file. To update integration settings in the future, go to the Ranorex menu bar and choose Tools > Jira > Configure integration.

For example, an especially helpful feature in Ranorex is that you can create a video recording of a test run. You can change the integration to attach the video report to the Jira issue. To enable the video reporting feature, go to the Ranorex Studio menu bar and choose Test Suite > Properties, then click the Report tab. Here, you can configure the report in a number of ways, such as changing the name or choosing a JUnit-compatible copy. The video reporting feature is off by default, but you can enable it either for all test cases, or for failed cases only.

Opening a new Jira issue for a failed test case

To test whether the integration works properly, configure a test case with a validation step that you know will fail. Next, execute the test, and confirm that the Jira issue is created in the way that you expected.

After confirming the integration with a manual test, you can switch to automatic issue handling. To do so, follow these steps:

1. From the Ranorex menu, choose Tools > Jira Integration > Configure.
2. On the Manual Issue Handling tab, turn off Enable issue buttons in report.
3. Switch to the Automatic issue handling tab and toggle on the option to Enable automatic handling, as shown below. (Note: you could have both automatic and manual on at the same time, but there isn’t really a benefit to doing this.)

It’s also recommended that you check the Upload video option to attach the Ranorex video report to the Jira issue. You can also change the summary and description, priority, labels, and more. However, be sure to leave the automatic identifier in the Summary field unchanged. 

Note: if you encounter issues with uploading video based on the maximum attachment size, you can edit the bitrate of the video reporting feature of Ranorex Studio to reduce the size. But because each test case gets its own video, these files do stay relatively small.

With automatic issue creation enabled, the test report in Ranorex will contain a link to the issue in Jira. Click the link to jump into your Jira backlog and view the issue.

Updating a Jira issue when the linked test case succeeds

Ranorex Studio connects a Jira issue to a test case through a link. Even if you rename or move a tests, the relationship between the Jira issue and the Ranorex Studio test is maintained.

When the test executes successfully, the corresponding Jira issue is closed automatically, as shown in the test run report below.

The integration between Jira and Ranorex Studio delivers instant feedback on the health of your application. Ranorex also integrates very well with TestRail for test case management, or you can use any other test case management solution that integrates with Jira.

To experience the integration for yourself, start a free, 30-day trial of Ranorex Studio today.

Gone are the days when developers would build a product for six months and then finally push it to testers to validate it. The obvious reason teams had to change this approach was that finding defects late in the development process and fixing them proved to be wasteful in time, effort and cost. Now, it is all about finding defects as early as possible and releasing products in smaller increments. This helps to deliver features at a faster rate and get quicker feedback.

One approach that has helped speed up this process is implementing a CI/CD pipeline. Although it may take considerable effort to enact a CI/CD transformation, the benefits outweigh the initial effort it takes. CI/CD gives the freedom to push any change without being afraid of breaking anything. We have different points where we can catch defects and fix them immediately. Features are developed incrementally, tested immediately and deployed automatically.

The availability of version control tools like Assembla, Docker containers, and feature flag tools that can turn on or off some portion of the code at any point in the CI process gives teams the flexibility to have production-ready code at any instant.

In addition to faster deployments, here are five additional benefits from implementing a CI/CD pipeline that you might not have expected.

1. Reduced risk

Testing and deploying code frequently helps to reduce the risk level of the project. You detect bugs as early and as fast as possible and can take immediate action as soon as a defect is found. This reduces the cost of fixing them, aligning with the shift-left paradigm.

Another advantage against risk is that teams get visibility into what happens throughout the development process, as automated tests get triggered in each stage of the development process.

Finally, developers and testers can set standards and ensure the code adheres to them by establishing different processes and tools. Overall, CI/CD helps to reduce risks considerably.

2. Better collaboration

In a CI/CD pipeline environment, all roles, including developers, testers, business representatives and infrastructure teams, must work together due to the interdependencies between their departments’ work. This helps to increase communication and fosters better collaboration among teams. It also helps everyone get insights into the work everybody is doing and how it all aligns with the project’s goals.

3. Less manual effort

For an effective CI/CD pipeline implementation, we need automation right from the start — when you check in code, trigger tests and deploy the application. At first, it takes time to set up all these automated tests, but the ROI on the amount of time you save on manual effort is exponential once the setup is complete.

Can you imagine trying to build, test and deploy all your code manually? This approach would never work in the current era, where organizations deploy builds to production multiple times a day. While automated tests for routine, repeatable tasks get triggered at different stages of the pipeline, developers and testers can focus their efforts on other areas that need more critical thinking and manual effort, like coming up with new ways to implement code, ideas for new features, building utilities that help to deliver products faster, and much more.

4. More extensive logs

A useful part of having a CI/CD pipeline is the data that you can get from each stage of the development process. Teams get extensive logging information at each step of the process, which helps them understand code better and aids in faster troubleshooting as soon as problems are discovered.

Users get notified if something goes wrong in the pipeline via alerts from integrated tools and email. These logs help for audit purposes and also ensure no issues are happening underneath the hood.

5. Easier rollbacks

A great beauty of having a CI/CD pipeline is that you can roll back changes quickly. If any new code changes break the application, you can immediately return it to its previous state within minutes.

Conclusion

As the world shifts toward faster release cycles to meet growing customer demands, deploying quality code multiple times a day is no longer a luxury; it has become a necessity. If your organization wants to stay competitive in this growing market, a CI/CD implementation shift may just be the right approach. The initial groundwork may be time-consuming, but the benefits are worth the effort. It is about planning for the future and playing the long-term game.