Return

Pattern: No Long Tests in CI/CD

Execute non-critical long-running tests in the background so they don’t block delivery to production

CI/CD is in place, and most tests are automated. Some are taking hours or even longer, and others require manual execution.

In This Context

Delivering changes quickly is a major goal of the cloud native approach to delivering software.Long-running performance tests, reliability tests, and manual and other types of full-system tests can take too long and delay delivery for hours—rendering CI/CD less valuable. Instead of dozens or even hundreds of times a day, delivery frequency gets reduced to just a few times per day. Similarly, fixing a bug or problem goes from taking a few minutes to instead requiring several hours.
• Manual intervention can create very long delays.
• Frequent integrations require a fast build/test cycle.

Therefore

Run your fastest tests earliest in the process. Schedule all tests that are manual, or which take longer than a few minutes to run, outside of the normal delivery process.If you have a test that is critical for functionality, however, it should be run asa blocking test.Mitigating risk is an equally important cloud native goal, which is why automated testing is a core tenet of the architecture. These two things need not conflict. There are strategies that allow adequate testing while enabling teams to still release quickly and constantly. Short-running tests can be incorporated into the pre-deployment process, while long-running tests can be executed in the background without blocking delivery to production.
• Run tests periodically post-delivery, and if a problem is found, either roll back the change or fix the problem and roll forward in a new release.
• Run long tests in parallel.
• Split the test automation into smaller test segments.

Consequently

Testing does not delay or disturb delivery. Quality of the products is kept high by the right balance of ever-changing tests.
+ Non-blocking long-running tests reveal problems without slowing velocity.
+ Release is quick and easy.+ Devs can deliver many times a day.
- Some issues can carry through to production.
- Requires strong roll back/roll forward protocol and procedures in place.