In software development there is a typical problem: how to maintain a relevant version of the test suite for each version of the project?
The goal for every version of project is to have a test suite version that provides accurate feature coverage. Let’s explore this idea in further depth.
Test Suite Scenario
Let’s imagine a simple model when you have version 2.0.0 of your project released to the public and version 2.1.0 is currently under active development. The test suite lives in its own separate repo.
Let’s say you have 50 test cases to cover all the functionality of 2.0.0 project version and test suite version is 15.2.0. Seems like the test team is active. For version 2.1.0 of the project test team added 5 new test cases and versioned test suite as 16.0.0.
Now let’s imagine customers reported a critical bug that development team quickly fixes in the version 2.0.1. The test team have to update test suite as well. Updates were made and version 15.2.1 of the test suite was released. In other words version 15.2.1 of the test suite provides appropriate feature coverage for version 2.0.1 of the project.
As the project evolves, it becomes difficult to tell which version of test suite should be used for specific version of project. And here we’ve described quite simple example.
Reality looks much different. It’s very common in large enterprises to see multiple releases being developed simultaneously and project is relying on components developed by separate teams.
Let’s take hybrid mobile app as an example. The test matrix would include mobile app version, web app version, mobile platform and device model, environment setup (which is a beast by itself). Mobile app changes, the web app changes. Some users use iOS device and some are in love with Android. Screen size varies. Environment can be loaded with various versions of database, different legal and compliance materials. You’re getting the picture.
Test Suite Synchronization
My previous project test team used branching strategy to assist with this issue. There were three environments: QA, STAGE, and PROD. There were four branches in test suite repo: master, qa, stage, and prod. Current development for new tests and any refactoring was done on the master branch. Other three branches where design to carry test suite version that is appropriate for project version that is currently deployed to specific environment. As build is passing validation in each environment, test code would be merged to next higher environment branch.
This branching strategy was taking care of some of axes in the test matrix we’ve described earlier. Web app, DB, legal and compliance versioning were covered. Mobile platform, OS version and device permutations were handled in the code by introduction of if-else-case statements that would point to correct block of code. Not an ideal solution.
On our current project, we’ve decided to merge test suite and project together. Codebase for both would live in the same repo and will have the same version. If any changes are done to the project – version is bumped for both.
The project is set up as a Maven project.
In order to accomplish the model described above, an extra <module> was added to root pom.xml to achieve aggregation. Which means when Maven goal is invoked at the root level it will be run against each individual module. Also each module’s pom.xml have <parent> section to reference root pom.xml as a parent pom. This means settings like <groupId>, <version>, <distributionManagement> and others are shared by the root pom with modules.
In our case we wanted to accomplish two goals when changes are done to the project codebase:
- global version update
- simultaneous artifact creation and deployment for both project and the test suite
In order to update version for root pom and all modules that are specified in the pom.xml, we need to use mvn version:set -DnewVersion=<new_version_here> goal at the root level.
One of the things our team have done to transform delivery process at the client location was to implement software delivery pipeline.
On Jenkins we would have build, deploy, smoke test and regression test jobs. These jobs are chained and would trigger one another in order from left to right.
Build job would pull project repo from BitBucket and invoke mvn clean deploy on the root level. This would not only build the project but also run unit tests and deploy artifacts to Artifactory. After that it would run registerVersionWithUCD.groovy script that would instruct UrbanCode Deploy to create version under respective component and store URLs for the artifacts deployed to Artifactory.
For our use case we wanted so that smoke test job would have workspace populated only with test suite related files.
To accomplish this, we’ve taken these actions:
- Added assembly plugin to <build> section of the test suite pom.xml to zip the whole folder structure during mvn deploy invocation
- Updated registerVersionWithUCD.groovy script to register version in UCD with test suite artifact URL from Artifactory
- Created getTestSuite.groovy script what would pull test suite zip-archive from Artifactory using the above URL and unzip it to current directory (job’s workspace)
At that point smoke test job would simply need to invoke mvn test goal to run smoke tests.
What Have We Achieved with This Setup?
Now it’s easy to tell which version of test suite we need to use for which version of project. As long as developers follow protocol and bump version using mvn version:set it would propagate to the test suite pom.xml and will be identical to version in root pom.xml. And there are number of automated ways to help with this as well.
That’s good by itself but benefits of merging project and test suite in one repo did not stop there.
This model encouraged developers and testers truly come together and start to collaborate.
Both developers and testers without any additional incentive started to review each other pull requests and familiarizing themselves with codebase out of pure curiosity.
All of the sudden, new features were implemented with tests in mind, testers were promoting quality right from the pull requests phase catching small and big problems early. During sprint planning meetings developers started to ask questions like: “what do you need from me to enable you to create tests for this?”
Furthermore members of these teams started to hang out together and feel happier at workplace.
Now that’s a true DevOps Transformation.
Liatrio is a collaborative DevOps consulting firm that helps enterprises drive innovation, expedite world-class software delivery and evolve their culture, tools and processes.
We work as “boots on the ground change agents,” helping our clients improve their development practices, get better at delivering value from conception to deployment and react more quickly to market changes. Our tech veterans have the experience needed to execute on DevOps philosophies and Continuous Integration (CI) and Continuous Delivery (CD) core practices, including automation, sharing, and feedback. Through proven tools and processes and a collaborative culture of shared responsibilities, we unite our clients’ technology organizations to help them become high-performing teams. Reach out — let’s start the conversation.
Liatrio is also hiring! If you want to be a part of a team that solves challenges around software delivery automation, deployment pipelines and large-scale transformations, contact us!