Skip to main content

Continuous Testing On Our Pipelines

I’m going to assume that I’m not the only one who has had hours of discussions around how much automation to plug in to our pipelines.

Firstly, I want to briefly talk about different environments and what I believe they should be used for. I’m going to keep this high level as I don’t want to make an assumption on what environments your pipeline will consist of.

Each of these could be made up of multiple environments. For example, ‘Test’ could suggest you have Local Dev, Integration and System Test. ‘Live’ could suggest you have Client Test, Live and Disaster Recover, but they would fall in to the categories I have illustrated. 

Test Environment(s)

I see test as being the environment where all new functionality is tested whether that be isolated components or end to end. Testing can be automated where appropriate and added to the products over all regression pack. Accept that in test you may need to apply an element of mocking as it’s not always possible to test external third parties from these types of environments. When deploying to test id be keen to run all automated tests as regression on the back of our automated deployment process, this allows the tester to concentrate on new functionality.  


I always ask product teams a question, what is the purpose of a Pre-Production environment?

For me it’s a pre-live rehearsal. We want to prove we can successfully deploy to a live like environment. (Ideally Pre-Production infrastructure should mirror Production where possible). There may be some other aspects like third party integration checks where you have had to mock up in test, this can be automated.

Should you run all tests in Pre-Production? Probably not, if you are not confident in the software at this stage it should never have left the test environments.


OK, so now you are releasing, whether this be a release to a client test environment for their own acceptance testing, canary release or becoming generally available. We still need to ensure the deployment was a success before anyone gets their hands on it for the first time.

For both Pre-Production and Production environments should we stop and automate all of our testing right now?

No. Start by writing a couple of automated tests - the ones that validate the most important functionality in the system. Get those running on every commit? Then the rule is to add new tests to cover new functionality that is added, and functionality that is being changed. Over time, you will evolve a comprehensive suite of automated tests. In general, it’s better to have 20 tests that run quickly and are trusted by the team than 2,000 tests that are flaky and constantly failing and which are ignored by the team. We should have regular reviews of these tests to ensure they are still adding value to the deployment.

Does continuous delivery mean firing all our testers?

Not at all. Testers have a unique perspective on the system - they aim understand how users interact with it. I recommend having testers pair alongside developers (in person) to help them create and evolve the suites of automated tests. This way, developers get to understand the tester’s perspective, and testers can start to learn test automation if required. Testers should also be performing exploratory testing continuously as part of their work. Certainly, testers will have to learn new skills - but that is true of anybody working in our industry.

Manual activities are still important, such as exploratory testing and usability testing (although automation can help with these activities, it can’t replace people). We should be aiming to bring all test activities, including other elements such as security testing, into the development process and performing them continually from the beginning of the software delivery life-cycle for every product and service we build.


Popular posts from this blog

My Testbash Brighton 2017 Notes

This was my second Testbash... if you ever get the chance in the future these conferences are a must!
I took copious amounts of notes from the 9 talks and tried to highlight my key takeaways here... hope they make sense but please comment if you have any questions :-)
Amy Phillips - Continuous Delivery
A survival guide to joining a fast paced environment/project…
Where does testing fit within Continuous delivery:

As highlighted, basically from start to finish…
There are lots of things we can do when joining a project that is using Continuous Delivery but one of the main points from this talk was to do your research! There should be an element of "Continuous" in every aspect of the project.
·Learning the ling, what's the difference Continuous Delivery, Continuous Deployment, Continuous Integration, Continuous design, Continuous Improvements etc.? ·Understand what your role in the project is going to be ·Understand the teams values, What's going well and maybe what the team p…

Leeds Testing Atelier - April 2018

The Importance of Building a Good Community of Practice

Within every organisation i think we can all agree that 'collaboration' is a major factor to the success of any project or initiative.

As organisations grow, new teams are created and existing teams fragment from each other, this is natural and i doubt that will change.

So what impact does that have to an organisation?
Stress in the work place? Unmet expectations? Relationship breakdown? Low Morale? Dissatisfied Clients? Collateral damage is inevitable.
When clients are dissatisfied, they often take their business elsewhere, which costs your company money. Poor communication can lead to high employee turnover, which creates a cost of hiring and training for new positions. At the least, with lower productivity and an unclear sense of purpose, poor communication causes employers to pay for work hours that are not efficiently spent, costing money, affecting efficiency, and keeping employees from reaching their true potential.
Its sometimes difficult to work on projects where you/y…