Wednesday, 27 September 2017

Continuous Testing On Our Pipelines

I’m going to assume that I’m not the only one who has had hours of discussions around how much automation to plug in to our pipelines.

Firstly, I want to briefly talk about different environments and what I believe they should be used for. I’m going to keep this high level as I don’t want to make an assumption on what environments your pipeline will consist of.

Each of these could be made up of multiple environments. For example, ‘Test’ could suggest you have Local Dev, Integration and System Test. ‘Live’ could suggest you have Client Test, Live and Disaster Recover, but they would fall in to the categories I have illustrated. 

Test Environment(s)

I see test as being the environment where all new functionality is tested whether that be isolated components or end to end. Testing can be automated where appropriate and added to the products over all regression pack. Accept that in test you may need to apply an element of mocking as it’s not always possible to test external third parties from these types of environments. When deploying to test id be keen to run all automated tests as regression on the back of our automated deployment process, this allows the tester to concentrate on new functionality.  


I always ask product teams a question, what is the purpose of a Pre-Production environment?

For me it’s a pre-live rehearsal. We want to prove we can successfully deploy to a live like environment. (Ideally Pre-Production infrastructure should mirror Production where possible). There may be some other aspects like third party integration checks where you have had to mock up in test, this can be automated.

Should you run all tests in Pre-Production? Probably not, if you are not confident in the software at this stage it should never have left the test environments.


OK, so now you are releasing, whether this be a release to a client test environment for their own acceptance testing, canary release or becoming generally available. We still need to ensure the deployment was a success before anyone gets their hands on it for the first time.

For both Pre-Production and Production environments should we stop and automate all of our testing right now?

No. Start by writing a couple of automated tests - the ones that validate the most important functionality in the system. Get those running on every commit? Then the rule is to add new tests to cover new functionality that is added, and functionality that is being changed. Over time, you will evolve a comprehensive suite of automated tests. In general, it’s better to have 20 tests that run quickly and are trusted by the team than 2,000 tests that are flaky and constantly failing and which are ignored by the team. We should have regular reviews of these tests to ensure they are still adding value to the deployment.

Does continuous delivery mean firing all our testers?

Not at all. Testers have a unique perspective on the system - they aim understand how users interact with it. I recommend having testers pair alongside developers (in person) to help them create and evolve the suites of automated tests. This way, developers get to understand the tester’s perspective, and testers can start to learn test automation if required. Testers should also be performing exploratory testing continuously as part of their work. Certainly, testers will have to learn new skills - but that is true of anybody working in our industry.

Manual activities are still important, such as exploratory testing and usability testing (although automation can help with these activities, it can’t replace people). We should be aiming to bring all test activities, including other elements such as security testing, into the development process and performing them continually from the beginning of the software delivery life-cycle for every product and service we build.

No comments:

Post a Comment