Skip to main content

Continuous Testing On Our Pipelines

I’m going to assume that I’m not the only one who has had hours of discussions around how much automation to plug in to our pipelines.

Firstly, I want to briefly talk about different environments and what I believe they should be used for. I’m going to keep this high level as I don’t want to make an assumption on what environments your pipeline will consist of.

Each of these could be made up of multiple environments. For example, ‘Test’ could suggest you have Local Dev, Integration and System Test. ‘Live’ could suggest you have Client Test, Live and Disaster Recover, but they would fall in to the categories I have illustrated. 

Test Environment(s)

I see test as being the environment where all new functionality is tested whether that be isolated components or end to end. Testing can be automated where appropriate and added to the products over all regression pack. Accept that in test you may need to apply an element of mocking as it’s not always possible to test external third parties from these types of environments. When deploying to test id be keen to run all automated tests as regression on the back of our automated deployment process, this allows the tester to concentrate on new functionality.  


I always ask product teams a question, what is the purpose of a Pre-Production environment?

For me it’s a pre-live rehearsal. We want to prove we can successfully deploy to a live like environment. (Ideally Pre-Production infrastructure should mirror Production where possible). There may be some other aspects like third party integration checks where you have had to mock up in test, this can be automated.

Should you run all tests in Pre-Production? Probably not, if you are not confident in the software at this stage it should never have left the test environments.


OK, so now you are releasing, whether this be a release to a client test environment for their own acceptance testing, canary release or becoming generally available. We still need to ensure the deployment was a success before anyone gets their hands on it for the first time.

For both Pre-Production and Production environments should we stop and automate all of our testing right now?

No. Start by writing a couple of automated tests - the ones that validate the most important functionality in the system. Get those running on every commit? Then the rule is to add new tests to cover new functionality that is added, and functionality that is being changed. Over time, you will evolve a comprehensive suite of automated tests. In general, it’s better to have 20 tests that run quickly and are trusted by the team than 2,000 tests that are flaky and constantly failing and which are ignored by the team. We should have regular reviews of these tests to ensure they are still adding value to the deployment.

Does continuous delivery mean firing all our testers?

Not at all. Testers have a unique perspective on the system - they aim understand how users interact with it. I recommend having testers pair alongside developers (in person) to help them create and evolve the suites of automated tests. This way, developers get to understand the tester’s perspective, and testers can start to learn test automation if required. Testers should also be performing exploratory testing continuously as part of their work. Certainly, testers will have to learn new skills - but that is true of anybody working in our industry.

Manual activities are still important, such as exploratory testing and usability testing (although automation can help with these activities, it can’t replace people). We should be aiming to bring all test activities, including other elements such as security testing, into the development process and performing them continually from the beginning of the software delivery life-cycle for every product and service we build.


Popular posts from this blog

My Testbash Brighton 2017 Notes

This was my second Testbash... if you ever get the chance in the future these conferences are a must!
I took copious amounts of notes from the 9 talks and tried to highlight my key takeaways here... hope they make sense but please comment if you have any questions :-)
Amy Phillips - Continuous Delivery
A survival guide to joining a fast paced environment/project…
Where does testing fit within Continuous delivery:

As highlighted, basically from start to finish…
There are lots of things we can do when joining a project that is using Continuous Delivery but one of the main points from this talk was to do your research! There should be an element of "Continuous" in every aspect of the project.
·Learning the ling, what's the difference Continuous Delivery, Continuous Deployment, Continuous Integration, Continuous design, Continuous Improvements etc.? ·Understand what your role in the project is going to be ·Understand the teams values, What's going well and maybe what the team p…

Is TestBash an Invaluable experience?

Over the last few years, I’ve attended four single track conferences at TestBash Brighton and Manchester. Most recently, September 2018 and April 2019.

One of the most common questions I get after the conference is, Was it worth it? Especially now given, I’m a contractor, and I attend at my own expense.

In short, Yes It’s definitely worth the trip!

That leads me into my question; does a single-track conference work for me, or could I get more out of TestBash?

Of course, there are pros and cons, some talks may not be as engaging, but this is no disrespect to the speakers, for me, it’s because I probably favour certain subjects and I’m pretty sure others will think the same.

Do I make the most of what’s on offer at TestBash? It pains me to say, No! I don’t make the most of the events leading up to and post the single-track conference so I could get more from the week.

TestBash offers master classes on automation. A full day of workshops. Test.bash() with talks on automation and technica…

The way we work shouldn’t be ‘Alien’ anymore, Should it!?!?

It’s been a good while since I have written anything (Mainly because I’m not that good a writer) but given my 16 months in the public sector I thought it was time… 
For a good number of years now the DevOps trend has given rise of buzzwords and methods aiming to speed delivery and it’s now an assumption of mine that most organisations are au fait with the term ‘Continuous Delivery’.
I was wrong…
Within the organisation that I am currently placed I believe I have a high-functioning collaborative team who work flexibly and in continuous cycle. We also involve the users which is a huge benefit to the development teams, but that’s not saying we get everything right 100% of the time! Teams must continue to learn and evolve to remain productive.
Our team work in one of several delivery groups and with this comes multiple challenges; Silos No single environment strategy lack of communicationsVaried ways of working.  And thats just to name a few, which is interesting when all delivery groups …